Let's be honest – the vast majority of software engineers are numbers people. What they're good at – and indeed what they earn their money doing – is creating solutions out of data, programmes and figures, not dealing with nuances and interpretation of language. But when it comes to the testing of medical device software, language is important, in particular the distinction between 'validation' and 'verification'.
The US Food & Drug Administration (FDA) in its 2002 General Principles of Software Validation points out that many software engineering articles and textbooks use the terms verification and validation interchangeably. Alternatively, software verification, validation and testing (known as VV&T) are commonly seen as a single concept, with no distinction made between the three terms.
Go deeper with GlobalData
But, while the FDA does make it clear in its guidance that these are, and should be treated as, separate and distinct terms, it does not emphasise this difference, meaning that, for many firms creating medical device software, particularly newcomers to what is a burgeoning industry, what is in fact a very important distinction can sometimes get lost.
"The document does not do a good job of differentiating actual verification and validation activities," warns medical device software engineer Robert Nadler. "It just calls everything validation."
The FDA, to be fair, in the guidance defines software verification as being the process that "provides objective evidence that the design outputs of a particular phase of the software development life cycle meet all of the specified requirements for that phase". Software validation, by comparison, it argues "is a part of the design validation for a finished device, but is not separately defined in the Quality System regulation".
Therefore, for purposes of the General Principles of Software Validation, the FDA argues software validation to be the "confirmation by examination and provision of objective evidence that software specifications conform to user needs and intended uses, and that the particular requirements implemented through software can be consistently fulfilled".
How well do you really know your competitors?
Access the most comprehensive Company Profiles on the market, powered by GlobalData. Save hours of research. Gain competitive edge.
Thank you!
Your download email will arrive shortly
Not ready to buy yet? Download a free sample
We are confident about the unique quality of our Company Profiles. However, we want you to make the most beneficial decision for your business, so we offer a free sample that you can download by submitting the below form
By GlobalDataKey differences between validation and verification
Verification tests whether: a product delivers the required functionality to the customer.
Validation tests whether: the functionality of a product, as defined through verification, is the intended behaviour of that product.
Testing for verification typically requires: reviews and evaluation of documentation and plans, evaluation of any codes and specifications, running through of checklists, walk-throughs of the software and inspection meetings. Verification will normally take place before validation, not vice versa.
Testing for validation typically requires: actual testing of the software itself, normally once any verification process has been completed.
Objective evidence for software testing
In more detail, the FDA argues that software verification is the process of providing objective evidence that the design outputs of a particular phase of the software development lifecycle meet all of the specified requirements for that phase. Software verification, it adds, looks for consistency, completeness, and correctness of the software and its supporting documentation, as it is being developed, and provides support for a subsequent conclusion that software is validated.
Software testing is one of many verification activities, as are static and dynamic analyses, code and document inspections and walkthroughs, among other techniques. What the FDA means, through its definition of software validation above, by contrast, is that software validation activities may occur during, as well as at the end of the software development lifecycle to ensure that all requirements have been fulfilled.
Since software is usually part of a larger hardware system, the validation of software typically includes evidence that all software requirements have been implemented correctly and completely, and that they are traceable to system requirements. A conclusion that software is validated, stresses the FDA, is highly dependent on comprehensive software testing, inspections, analyses and other verification tasks performed at each stage of the software development lifecycle.
"Testing of device software functionality in a simulated use environment, and user site testing are typically included as components of an overall design validation program for a software automated device," it emphasises.
However, the FDA is realistic enough to recognise that a developer cannot test forever. In large measure, therefore, software validation is a matter of developing a "level of confidence" that the device meets all requirements and user expectations for the software automated functions and features of the device. Measures such as defects found in specifications documents, estimates of defects remaining, testing coverage, among other techniques, are all therefore integral to developing an acceptable level of confidence before shipping the product.
But, stresses Nadler, this is a nuance that is more often than not lost on software developers. Software vocabulary is constantly changing and, certainly, has changed a lot since the 2002 publication of the guidance.
The issue, he argues, is much more important than an arcane debate over semantics. It is about communicating to developers that software testing needs to incorporate both elements: verification and validation. "The fact that the FDA does not particularly distinguish between validation and verification within its syntax is not really the issue," he explains. "The issue is to create a distinction between the intent of the testing, so it is clear what it is you are supposed to be doing in terms of verification and validation.
"The problem is that for engineers or quality systems people putting together procedures, you have to look very closely at what the intended process is rather than the words they are using. It is all a question of interpretation. It is not that the FDA needs to rewrite its documents, what is needed really is more of a distinction between the documents," he adds.
"Verification is answering the question of whether the engineer has built the product right, but that does not answer the question of whether or not the right product has been built, which is completely different. You can build the right product according to the specification but you still have to ask the question whether it is the right product, and the testing requirement for these questions will be different.
"A medical device software developer, for example, has to put together tests that will answer those questions. So it is about answering 'what do you need the product to do?' as well as whether it meets all the regulatory specifications."
Nadler explains that the biggest problem in the lack of distinction between these two phrases is for those companies that have not been in the medical device industry before and are coming into it. In developing software that may have a medical device application, companies have to be prepared to encounter and go through a level of scrutiny and testing that can be immense.
"Validation and verification are part of a much broader quality systems issue, an issue that has to be put into place company-wide," he explains. "If your company or organisation has not developed medical devices you will need to be putting in place the quality systems to answer both these questions and produce the documents that address all these issues. Verification and validation can, in this sense, be a very big issue.
"Putting in place validation and verification processes, particularly when you have not done it before, can be hard and you have to spend time on it and pay close attention to the details. You have to understand the validation and verification intent and not just get caught up in the vocabulary issues. The reality is that it is a much broader issue than just creating software for medical devices."
Regulatory hurdles and tough scrutiny
Anyone creating and developing medical device software needs to come into it with their eyes fully open and recognise the regulatory and oversight hurdles – including verification and validation – they will need to jump, Nadler says. "The scrutiny of software over the past 25 years has become much tougher," he concludes. "Just as the industry has changed, so the FDA has put more scrutiny on to it. Your models of software development need to take into account issues of validation and verification.
"We of course do not know where the FDA is going to go with respect to software testing in the future. Either way, software engineers need to be aware of the broader picture and, hopefully, the vocabulary that describes this will in time become more precise. If you are going to develop FDA-regulated software you have to have a quality system in place that you can follow and which covers these issues. You have to have high-quality, reliable software."
