This chapter starts with the existential crisis that companies face when hiring and retaining testers. Later sections provide a wider industry view and a proposed new skills set.
2.1 Existential Crisis for Testers
Testing Is Obsolete The general feeling was that the approaches offered by training providers, books and the certification scheme(s) are no longer fit for purpose. They are outdated and have not kept pace with the industry changes experienced by all members.
Replaced by Automation A common perception is that testers and testing in general can be replaced by automated approaches. Managers are swayed by the promise of Continuous Delivery (CD), pervasive automation and the emergence of machine learning and artificial intelligence. Testers have not found a way to articulate the reasons why testing is much harder to automate and eliminate than people believe.
How Do You Add Value to the Team? If you ask a tester what value they bring to their teams, they find it extremely difficult to make a strong case. The argument that testing is important is won already. But how do testers explain their value? My experience is that almost all testers don’t know who their customers (stakeholders) are; they don’t know what stakeholders want or how the information testers provide is used to make decisions still. As a profession (if testing actually is a profession) we have failed to make the case.
Titles Changing––Evolution of SDET Role Companies are implementing various ways of redistributing testing in their teams. The Shift-Left idea has caught on, with developers taking more responsibility, testers acting as coach/mentors to devs and other team members and being more closely involved in requirements. These are all good things. More popular in the US than Europe, the SDET (Software Development Engineer in Test) role is a hard one to fill. What is clear is that testing is in a state of flux. Testers are finding it hard to assimilate the change and to contribute towards it.
We’re All Engineers; Everyone Must Write Code Related to the SDET approach, testers who never wrote code (and might not ever want to) are being encouraged to learn a programming or scripting language and automated test execution tools. The pressure to program and use tools is significant. This is partly because of the relentless marketing of the vendors. But it is also fuelled by a lack of understanding of what tools can and should do and what they cannot and should
not do. Testers (whether they use tools or not) are not well briefed in the case for automation or the strategies for successful tool implementation.
Once Highly Respected Skillset/Mindset no Longer Valued The Year 2000 threat caused many companies to take testing seriously and there was a brief period when testers were more highly respected. But when Agile appeared on the scene and was widely adopted, the role of testers was badly or not defined. They were expected to just ‘get on with it’. Agile, at the start at least, was mostly driven as a developer initiative with little thought for how requirements and testing were done.
After 15 years, testers have a much better idea of their role in Agile. Unfortunately, the next big thing is Continuous Delivery. The mantra is that testing, of whatever type, should be automated. Once again, testers are under pressure to re-define their role and/or get out of the project.
Technology Changing at Unprecedented Rate There’s little doubt that test approaches have not kept pace with the changing technology. Although test execution tools appear within a year or two of new user interface technologies, the new risks, modelling approach and test management methods emerge very slowly. Tester skills seem to be tied to technologies. Skills should be independent of technology, enabling testers to testanything.
2.2 The Drive to Digital
Across the business world, there is a revolution in the way that IT is being specified, developed, implemented and used. There is lots of hype around the whole ‘Digital Transformation’ phenomena. Digital Transformation programs are affecting business across all industry and government sectors. There is no doubt that it also affects people in their daily lives.
Digital includes traditional IT but includes:
• Mobile anything
• The Internet of Things
• Autonomous vehicles
• Our home, workplace, public and private spaces
• Robots (physical)
• Bots (software)
• Artificial Intelligence, Machine Learning, Deep Learning
• And so on. . .
Digital visionaries promise a lot:
• No human intervention in your systems (Autonomous Business Models)
• Marketing messages created, sent, followed up and changed almost instantly
• Full range of data from the smallest locale to global in all media formats at your disposal
• Autonomous drones, trucks and cars can transport products, materials and people
• Physical products needn’t be ordered, held in stock and delivered at all––3D printing removes constraints.
Mankind has built some pretty impressive systems (of systems). The most complex systems ever used to be the Space Shuttle with 2.5 m parts, but this was superseded by the Nimitz class supercarrier which has one billion parts. In fact, the carrier comprises thousands of interconnected systems and with a crew of 5000–
6000, it could be compared to an average town––afloat.
Compare this with a ‘Smart City’. A Smart City is
an urban development vision to integrate multiple information and communication technol- ogy (ICT) and IoT solutions in a secure fashion to manage a city’s assets––the city’s assets include, but are not limited to, local departments’ information systems, schools, libraries, transportation systems, hospitals, power plants, water supply networks, waste management, law enforcement, and other community services. (Wikipedia)
With a smart city, the number of connected nodes and endpoints could range from a million to billions. The smart city will be bigger, more complex than anything before. Connected citizens and many of the systems:
• Move in the realm of the city and beyond
• Interact in unpredictable ways
• Are places where citizens are not hand-picked like the military; crooks, spies and terrorists can usually come and go as they please
Unlike ships, smart cities are highly vulnerable to attack.
Digital systems will have a social impact on all citizens who encounter them.
There are huge consequences as systems become more integrated with the fabric of society. Systems already monitor our every move, our buying, browsing and social activities. Bots push suggestions of what to buy, where to shop, who to meet, when to pay bills to us minute by minute.
Law enforcement is affected too. CCTV monitors traffic, people and asset movement and our behaviours. The goal might be to prevent crime by identifying suspicious behaviour and controlling the movement of law enforcement agents to places of high risk. But these systems have the potential to infringe our civil liberties and the legal frameworks are behind the technology.
Digital affects all industries.
Unlike Agile, which is an ongoing IT initiative, Digital is driven by Business.
Agile has taken more than 15 years to get halfway implemented in the software
industry. Digital has taken no time at all––perhaps 2–3 years and it is all-pervasive in the West. Digital is the buzz-phrase of the moment.
The speed of delivery is partly about pro-action, but it is also about survival.
Often, Chief Digital Officers are Marketers. Marketers move at the pace of marketing and they want change at the same pace. To the marketer, frequent software delivery is critical. Mobile users expect apps to change almost daily with new features, offers, opportunities appearing all the time. Users often don’t care which supplier they use, as long as their apps work reliably, so businesses are in an APPS RACE.
S/W development at the pace of marketing.
So, automation (and not just test automation) is critical. What business needs is IT responsiveness––what you might calltrue agility. This doesn’t necessarily mean hundreds of releases every day; but it does mean business want rapid, regular turnaround from ideas to software delivery.
With continuous integration/deployment, DevOps, developers can nowpromise Continuous Delivery
Testers need to provideContinuous Assurance
This means automation through the (shortened) life cycle. What exactly is possible and impossible with automation, right here, right now? Are Continuous Delivery and DevOps the route to success? Could testing be the bottleneck that prevents success? How do testers operate in dynamic, high-paced, automation- dominated environments?
2.3 Waterfall Thinking Won’t Work with Continuous Methods
Continuous Delivery or an adapted version of it is becoming increasingly popular in Digital projects and if Digital is the ‘future’ for the majority of organizations, then we had better prepare for it. Testers need to adapt to fit into their continuous delivery regimes so let’s look at how continuous approaches are normally described.
The most common diagram one sees is the figure eight or infinite loop below.
The principle is that the plan, code, build, test through release, deploy, operate and monitor phases are sequential but are repeated for every release.
But there’s a problem here. If you unwrap the infinite loop, you can see that the phases are very much like the stages of a Waterfall development. There are no feedback loops, you have to assume one phase completes before another starts.
So, it appears that Continuous Delivery is just Waterfall in the small. What do weknowabout waterfall-style developments?
• It’s sequential––one stage follows another––no variation
• Dependencies rule––you can’t start one stage before the previous stage is done
• It’s not re-entrant––no flexibility to react to external events
• Testing has stages itself––weknowthat testing has itself stages of thinking and activities spread through the process
• Only one phase of testing––but there are developer and tester manuals and automated test execution activities
• Testing is squeezed––timeboxed activities––the thinking, preparation and execu- tion time is all limited
• No feedback loop(s)––we know that testing finds bugs––but the continuous process has no feedback loop.
If Agile has taught us anything, it’s that the dependence on staged approaches made Waterfall unsuccessful in more dynamic environments.
Staged thinking won’t work in a continuous process.
We need another way of looking at process to make Continuous Delivery work.
2.4 Separating Thinking from Logistics
There are two problems to solve here:
1. The first is that there is no one true way or best practice approach to implement- ing, for example, continuous delivery. Everyone does it slightly differently, so any generic training scheme has to offer generic practices.
2. The second is that any credible training scheme must recognize that there are skills that can be taught in the classroom, but theemployermust take on the role of explaining local practices and embedding skills.
These local practices are what we calllogistics. Logistics are how a principled approach is applied locally. Locally might mean ‘across an entire organization’ or it could mean every project you work on works differently. If you work on multiple projects, therefore, you will have to adapt to different practices––even if you are working in the same team.
Principles and thinking are global; logistics are local.
It’s clear that to offer training alone is not enough. There must be a contribution by the local employer to nurture trainees, coach them in local practices and give them work that will embed the skills and local approach.
To offer training to practitioners, we must separate the principles and thinking processes from the logistics.
How do we do this?
2.5 Logistics: We Don’t Care
We need to think clearly and remove logistics from our thinking. The simplest way to do this is to identify the aspects of the local environment and descope them, so to speak. The way Paul usually introduces this concept is to identify the things that we don’t care about.
As a practitioner you will care deeply about logistics, but for the purposes of identifying the things that are universally applicable––principles and our thought process––we need to set them aside for the time being. Here are the key logistical aspects that we must remove ‘to clear our minds’.
Document or Not? We don’t care whether you document your tests or not.
Whether and how you record your tests is not germane to the testing thought process.
Automated or Manual? We don’t care whether you run tests by hand, so to speak, or use a tool, or use some form of magic. It isn’t relevant to the thought process.
The mechanism for test execution is a logistical choice.
Agile vs. Waterfall? We don’t care whether you are working in an Agile team or in a staged, waterfall project or are part of a team doing continuous delivery. It’s not relevant to the testing thought process.
This Business or That Business? We don’t care what business you are in whether it is banking or insurance or healthcare or telecoms or retail. It doesn’t matter.
This Technology vs. That Technology? We don’t care what technology you are working with. It’s just not relevant to the thought process.
Programmer or Tester? We don’t care who you are––developer, tester, user business analyst––the principles of testing are universal.
Test Manager or No Test Manager? We don’t care whether you are working alone or are part of a team, with or without a test manager overseeing the work.
This is a logistical choice, not relevant to the testing thought process.
2.6 Without Logistics: The New Model for Testing
If we dismiss all these logistics––what’s left? Some people might think we have abandoned everything, but we haven’t. If you set aside logistics, what’s left is what might be called the universal principles and the thought process. Now, you might think there are no universal principles. But there clearly are––they just aren’t muddied by local practices. Paul’s book,The Tester’s Pocketbook[4,5] identifies 16 so-calledTest Axioms.
Some Axioms, for example the stakeholder axiom, ‘Testing Needs Stakeholders’, are so fundamental they really are self-evident. Other axioms such as the Sequencing axiom, ‘Run our most valuable tests first––we may not have time to run them later’,
Fig. 1 The new model for testing
are more prosaic––it sounds logistical. But sequencing is a generally good thing to do––HOW you prioritize and sequence is your logistical choice.
The New Model for Testing is an attempt to identify the critical testing thought processes (Fig.1). A Webinar [5] and white paper [6] give a full explanation of the thinking behind the model, which is reproduced below.
The model doesn’t represent a process with activities, inputs, outputs, entry and exit criteria and procedures. Rather it represents themodes of thinkingwhich people who test go through to achieve their goals. Our brains are such wonderful modelling engines that we can be thinking in multiple modes at the same time and process thoughts in parallel. It might not be comfortable, but from time to time, we must do this.
The New Model suggests that our thinking is dynamic and event-driven, not staged. It seems like it could be a good model for testing dynamic and event-driven approaches like continuous delivery.
Using the New Model as the basis for thinking fits our new world of testing.
The ten thinking activities all have associated real activities (logistics usually) to implement them and if we can improve the way we think about the testing problem, we are better able to make informed choices of how we logistically achieve our goals.
There are several consequences of using the New Model. One aspect is how we think about status. The other is all about skills.
2.7 Rethinking Status
As a collaborative team, all members of the team must have a shared understanding of the status of, for example, features in the pipeline. Now, the feature may be placed
Fig. 2 Status is what we are thinking
somewhere on a Kanban or other board, but does the location on the board truly represent the status of the feature?
Consider the ‘three amigos’ of user, developer and tester. When it comes to agreeing status, it is possible that the user believes their job is done––the requirement is agreed. The developer might be writing code––it’s a work-in- progress. But the tester might say, ‘I have some outstanding challenges for the user to consider. There are some gaps in the story and some ambiguous scenarios we need to look at’ (Fig.2).
What is the status of the feature? Done? Work in Progress? Or under suspicion?
When all three participants are thinking the same way, then there is a consensus on the status of the feature.
Continuous Collaboration is an essential part of continuous delivery. The New Model provides a framework for meaningful discussion and consensus.
2.8 The (Current) Problem with Certification
The most prominent tester certification scheme is created and administered by the International Software Testing Qualifications Board––ISTQB [7]. The training classes run and qualifications awarded number 875,000+and 641,000, respectively.
There are some minor schemes which operate, but ISTQB has become the de facto standard in the industry.
But there are well-known problems with current certification:
• If you look at the certification scheme syllabuses (Foundation and Advances Test Analyst, for example), the table of contents comprises mostly what we have called logistics.The certification schemes teach many of the things we say we do not care about.
• The schemes mostly offer one way of implementing testing––they are somewhat aligned with various standards. Incident Management, Reviews, Test Planning, Management and Control are all prescriptive and largely based on the Waterfall approach.
• Much of the syllabus and course content is about remembering definitions.
• The syllabuses infer test design techniques are procedures, where the tester never models, or makes choices. The tester and exam-taker are spoon fed the answers rather than being encouraged to think for themselves. There is no modelling content.
• The syllabuses don’t teachthinking skills.The wordthinkingappears once in the 142 pages of Foundation and Advanced Test Analyst syllabuses.
• Exams, being multiple choice format, focus on remembering the syllabus content, rather than the competence of the tester.
Certification does not improve your ability to be independent thinkers, modelers or (pro-)active project participants; the exams do not assess your ability as a tester.
This is a big problem.