Usability of Medical Devices – Formative Evaluation and Summative Evaluation

What does usability mean?

The IEC 62366-1 provides a good definition for this question. According to it, usability describes the characteristics of the user interface that facilitate use and thereby enable effectiveness, efficiency, and user satisfaction in the intended use environment. The user interface is the part of the medical device with which the user interacts during use. The user environment is the environment in which the user operates the medical device.

The IEC 62366-1 provides a good definition for this question. According to it, usability describes the characteristics of the user interface that facilitate use and thereby enable effectiveness, efficiency, and user satisfaction in the intended use environment. The user interface is the part of the medical device with which the user interacts during use. The use environment is the setting in which the user operates the medical device.

Gebrauchstauglichkeit_in_der Medizin

How can it be determined if something is usable?

By definition, it is about demonstrating aspects of effectiveness, efficiency, and user satisfaction – but none of these are physically measurable quantities. So how do you proceed?

As is often the case in medical technology, a standard comes into play here – the already mentioned IEC 62366-1, to be precise. This standard specifies different steps of a usability engineering process that must be followed and that are linked to risk management according to ISO 14971. By applying the standard, usability can be demonstrated – up until the point when contradictory information becomes known. Such a situation may arise, for example, during post-market surveillance if it is found that there are difficulties with the usability of the product.

For which medical devices is usability relevant?

According to the MDR and IVDR, existing risks due to use errors must be reduced as far as possible. To achieve this, manufacturers must

  • “reduce the risks due to ergonomic features of the product and the environment in which the product is to be used as far as possible (patient-safety-oriented product design), as well as
  • “take into account the technical knowledge, experience, education and training, where applicable the use environment, as well as the health and physical condition of the intended users (product design oriented towards laypersons, professionals, persons with disabilities, or other users) [MDR / IVDR].”

The above-mentioned requirements are derived from the General Safety and Performance Requirements (often abbreviated as GSPRs, following the English terminology). Since the GSPRs apply to all medical devices, it must be assessed on a case-by-case basis whether the stated requirements are applicable and to what extent.

How is this related to risk management?

Usability is primarily about reducing risks from use errors as much as possible. The handling of risks of medical devices is regulated in ISO 14971, i.e., in risk management. The usability engineering process is intended to assess the user’s interactions with the user interface (user-product interface) with regard to potential risks. This also includes the information to be provided, such as the instructions for use and aspects related to:

  • Transport
  • Storage
  • Installation
  • Use
  • Maintenance and repair
  • Disposal

In addition, characteristics of the user interface that are related to the safety of the medical device are to be identified. This information is then fed into the risk management process.

Usability Process – Usability Engineering Process

The IEC 62366-1 standard provides the framework for a usability process as well as detailed information on the necessary steps. It even includes a flowchart that illustrates not only the process flow but also the interfaces with risk management. We recommend following these procedures and not attempting to streamline or modify the prescribed process.

How do you create the “Use Specification”?

The use specification is a detailed description of the application of the medical device. It is important to align this with the actual use in the clinical environment. To do so, the following questions must be considered:

  • For which indications is the medical device intended to be used?
  • Which patient group is intended to be treated?
  • On which parts or tissues of the human body is the product used?
  • Who are the intended users and what qualifications do they have?
  • In which environment is the product used?
  • How does the product work?

By answering these and, if necessary, additional questions, a clear picture emerges of how and where the product is used in the field. In practice, it can be difficult to answer some of these questions. This may be due, for example, to the fact that the environment in which the product is used can vary greatly. In some cases, users with different profiles are also conceivable. It is fundamentally important for a manufacturer to build a comprehensive understanding of their own product and its use in the field. While this may sound obvious, we increasingly observe during the transition to the MDR that some companies lack this knowledge about their own products. Only on the basis of this understanding is it possible to conduct solid risk management and usability engineering.

What are the safety-relevant characteristics of the user interface?

As part of risk management according to ISO 14971, the safety-relevant characteristics of the product must be determined. This also includes characteristics of the user interface that are related to safety. To identify these characteristics, the techniques of the usability engineering process can be used. Following the path through IEC 62366-1 is not mandatory. What is important is that the characteristics are determined. If there are product-specific standards, any information or requirements regarding the user interface contained therein must be taken into account.

But which characteristics of the user interface are safety-relevant? This is highly product-specific. To make this decision, you should first be aware that usability in the context of a medical device is not primarily about making the use of the product as convenient as possible. It is about specifically identifying and minimizing the risks associated with use. In fact, the most convenient use often even contradicts safe use!

But why is that? If a product is convenient to use, you can operate it without thinking. This means you do not consciously engage with its operation. That is fine as long as the product is not safety-critical. However, if you, for example, change a patient’s medication via their infusion pump without thinking about it, things can quickly become dangerous. The usability of medical devices ideally strikes a balance between an application that is both convenient and safe. Nevertheless, safety takes precedence over everything else!

Based on this, consider which parts of the user interface could lead to a hazard in the event of a use error.

What is a use error?

The standard requires that potential use errors be identified. This involves considering everything a user could do wrong while operating the user interface. A use error also occurs if the user is unable to perform or complete the medical device operation or a necessary task with it. In addition, there is no limitation regarding whether the user notices the use error or not.

By definition, a use error is a user action or lack of user action during the use of the medical device that leads to a different result than intended by the manufacturer or expected by the user.

Formative Evaluation / Development-Accompanying Evaluation

Formative evaluation is carried out during the design and implementation of the user interface in order to examine the user interface, identify the need for improvement, or confirm the adequacy of the user interface.

By definition, the aim is to identify strengths, weaknesses, and previously unknown use errors related to the user interface. In the development-accompanying evaluation (formative evaluation), there are usually no formal acceptance criteria. During these activities, the user interface is to be developed step by step in order to achieve a certain quality level and to increase the likelihood of passing the final evaluation (summative evaluation).

Formative evaluation is a multidisciplinary task that requires a team of people with different qualifications. To genuinely focus on and implement user requirements and needs, it should be considered to include not only engineers, psychologists, developers, and other experts, but also the intended users.

In principle, the user interface should be iteratively improved until the required quality level is assumed to be achieved. Once the manufacturer considers the development to be advanced enough to pass the final evaluation, further development of the user interface can be discontinued.

Summative Evaluation / Final Evaluation

The implemented user interface is subject to a final evaluation based on the testable requirements defined in the summative evaluation plan. This evaluation is carried out at the end of the user interface development with the intention of providing evidence that the user interface can be used safely.

Since the accompanying documentation is part of the user interface, it should be available for the final evaluation. At this stage, the aim is to simulate the use of the medical device as realistically as possible. This not only includes involving the intended users for the tests, but also providing the information that accompanies the medical device in practice. And, of course, simulating a realistic test environment, which, if necessary, also involves prior training of the users.

In the final evaluation, formal acceptance criteria must be met. These are defined beforehand during the planning phase and are intended to address the selected hazard-related use scenarios. At this point, ensure that the applied criteria are also consistent with your documented risk policy. The summative evaluation is part of the verification and validation of the product design and serves as a form of validation of the application-related safety aspects of the user interface. As is customary in validation, the same people who developed the user interface should not participate in the summative evaluation.