Qualitative Study on the Perceived Ease of Use of a
Knowledge Translation Website: www.strokengine.ca

Yasmine Chibane; Annie Rochette

Special Article – Stroke Rehabilitation

Phys Med Rehabil Int. 2020; 7(1): 1168.

Qualitative Study on the Perceived Ease of Use of a Knowledge Translation Website: www.strokengine.ca

Yasmine Chibane^1,2 and Annie Rochette^1,2*

¹School of Rehabilitation, University of Montreal, Montreal, Canada

²Centre for Interdisciplinary Research in Rehabilitation in greater Montreal (CRIR), Montreal, Canada

*Corresponding author: Annie Rochette, OT, PhD, School of Rehabilitation, Université de Montréal, C.P. 6128, Succursale Centre-Ville, Pavillon Parc, Montréal (Québec) H3C 3J7, Canada

Received: January 13, 2020; Accepted: March 06, 2020; Published: March 13, 2020

Abstract

Objective: To assess the perceived ease of use of Strokengine, a knowledge translation website on stroke rehabilitation by targeted users.

Methods: A qualitative approach gathering each participant’s opinion on the strengths and weaknesses while performing specific tasks (n=10), based on a standardised scenario. Ease of use was further informed by the Post Study System Usability Questionnaire. The data was analysed according to a theoretical thematic analysis.

Results: The eight participants (two clinicians, two students, two stroke survivors and two family members) were between 21 and 83 years of age. All but one relative indicated good satisfaction with the site, as the role of the relative was perceived as not being sufficiently highlighted. According to participants (7/8), the site is useful, fast, easy to learn and has a good quality interface and information. The organization of the interventions section (3/8 participants) makes it difficult to find information. For a family member and one student, the site should have fewer contrasts of colour and condensed text.

Conclusion: Overall, Strokengine is easy to use as it is practical for searching evidence-based information, fast and easy to learn.

Practice Implications: Results highlights the importance of periodically assessing ease of use of knowledge translation websites.

Keywords: Knowledge translation; Stroke; Usability; Website; Rehabilitation

Introduction

Knowledge transfer (KT) in health is a structured process of making the results of scientific research accessible to professional practice, policy development and the general public [1]. This is done to promote appropriate use of knowledge by potential users [2]. To be effective, KT involves the popularization of knowledge in accessible language and tools that can be used in real situations (e.g. website).

Websites with health as their main subject have been around since the 1990s and have become an important source of information for Internet users. While so-called scientific information was produced, disseminated and controlled by scientists and/or the state, now anyone with access to the Internet can produce this type of information [3]. Content may therefore be false and different issues must be considered (e.g. health, safety or well-being). To address this, evidence-based sites have emerged so that factual information (from research) is shared with potential users and follows current clinical practices [4].

The www.strokengine.ca site (called Info-AVC in French) is an example. Its mission is to make research data on stroke rehabilitation accessible to bridge the gap with current practices. It is intended for anyone who wants evidence-based information on stroke rehabilitation, specifically clinicians, stroke survivors and their families. Strokengine offers information from quality articles, websites and systematic reviews on the various assessment tools useful in stroke rehabilitation. This site also presents the level of evidence of the effectiveness of stroke rehabilitation interventions following a thorough review by the site team. More recently, it includes also e-learning resources such as e-aerobics online course for physiotherapists.

An essential quality measure when designing a website is its ease of use. This concept refers to the degree to which a product can be used by users to perform specific tasks effectively, efficiently and satisfactorily in a specific context [5]. ISO-9241 norm thus introduces three concepts when defining ease of use: effectiveness, efficiency and satisfaction [5]. Effectiveness refers to the ability of users to achieve given goals in specific environments (accurately and completely). Efficiency involves the resources/efforts deployed to achieve the goal. Rather, satisfaction is about the comfort and acceptability of the system to its users. Several other qualities have an impact on the usability of a website, including usefulness in relation to the task, accessibility to users, ease of learning and user safety [6]. A site with low usability would, for example be an easy to use site, but one that does not allow the user to perform a task adequately. As a result, a website with good usability is useful for the task at hand, accessible to users, secure, efficient (fast) and easy to learn. Since the application of the various principles in the literature related to ease of use does not guarantee a perfect site, the evaluation phase is essential [7].

In the scientific literature, two studies have been previously conducted in relation to the Strokengine site and its ease of use [8,9]. However, these are usability assessments of older versions/platforms of the site (2009 and 2012). Indeed, the Strokengine-Family site, the Strokengine for clinicians and the Strokengine-Assessment are now integrated into a single website that has another configuration. For this website, specifically, a usability assessment has not been conducted while users are currently looking for information on this site to guide them in their decision-making. Thus, the objective of this study was to assess and describe the perceived ease of use of Strokengine by website users in order to improve the website.

Methods

A qualitative approach [10,11] was adopted for the following study, which was approved by the Research Ethics Board of Institutions where recruitment took place.

The target sample was eight people: two clinicians, two students, two people who had had a stroke and two relatives of people who had had a stroke. This sample represents the users who are most likely to access the Strokengine site. Some authors indicate that at five users, 80% of website problems are detected [12,13]. Others, such as Turner and colleagues [14], indicate that a group of seven participants is optimal. Indeed, the majority of problems should be detected by first users and the chances of further problems being discovered decrease with additional users [14]. As a result, with eight participants, it was expected that the majority of the site’s problems would be detected.

Clinicians practicing with a stroke clientele, on a part-time or full-time basis in an affiliated rehabilitation center were recruited by e-mail, as were students in the professional master’s degree in occupational therapy from an affiliated University in Quebec, Canada. People who had a stroke in the last five years (era of medical technology) and relatives of patients with stroke were recruited either from the Department of Physical Medicine and Rheumatology located in Belgium or from the Canadian Heart and Stroke Foundation. Participants had to be able to understand and speak French. Persons with significant cognitive or phasic impairments based on clinical judgment preventing informed consent were excluded.

In this study, the informal approach to ease-of-use testing was prioritized. Thus, instead of an expert evaluating the site, participants had to use the site by performing specific tasks presented in the form of instructions [15]. Throughout the process, the evaluator asked the participant to talk out loud. This approach made it possible to highlight serious and recurring problems when using the site while measuring the impact of problems on users and thus establishing a priority for problem resolution. For example, by giving a specific task to the user to find a specific assessment on the site made it possible to determine if the user can find the assessment without difficulty. If there were any difficulties, they were noted and then analyzed. Realtime observation determined whether this was due to factors specific to the individual (e.g., lack of experience with Internet use) or rather external factors such as the readability of the characters on the site or the visual organization of the site.

Data collection was based on the methodological approaches for the evaluation of health information systems [16]. The aim was to observe users’ interactions with the site, observe their performance and record their preferences through a scenario lasting a maximum of thirty minutes. The scenario comprised ten varied tasks such as accessing the strokengine website in French, finding all assessments relating to driving post-stroke, finding a list of all interventions or finding a way to contact the team. To observe users’ interactions with the site, data collection grids were used to record important information (e.g. general comments, errors made, slowdowns, user irritations or expressed wishes). Participants were filmed during the process for any additional information during the analysis of the results. Following the completion of the scenario, the evaluator completed with the user a free translation of the Post Study System Usability Questionnaire (PSSUQ) [17] to assess the participant’s satisfaction with the use of the website. The questionnaire was used in an exploratory and complementary manner to qualitative data collection. These are 19 questions with answers on a Likert-type scale (7 points) where the lower the answer, the higher the participant’s satisfaction with the site. Following each question, there is a space provided for comments to clarify/expand the participant’s answers. The PSSUQ can be used to produce a mean measurement of the user’s overall satisfaction with the site (questions 1-19) as well as measures of three sub-scales: system utility (questions 1-8), information quality (questions 9-15) and interface quality (questions 16-18). It thus makes it possible to evaluate the participant’s experience when using the website. This questionnaire has excellent reliability at the level of the three subscales (0.91 to 0.96). As for validity, the user’s overall satisfaction with the site, as well as the usefulness of the system and the quality of the interface, are significantly correlated with the success rate of the scenarios. This questionnaire is also sensitive to differences between subscales [18].

Each participant was met once for a maximum of one hour. The profile of each user was detailed, by interview, which included sociodemographic data (gender, age, category of participant) as well as data to describe their knowledge and experience with a computer. Indeed, user’s knowledge is essential in order to make assumptions about performance and difficulties encountered. Each participant had to perform the tasks detailed in the scenario. The participant was informed of the tasks to be performed and had to estimate the time required to complete all tasks to the nearest minute. The scenario was designed to assess the efficiency, effectiveness and satisfaction of participants with the use of the site. The scenario consisted of tasks that users are likely to perform [15]. All users had to perform the same scenario. Indeed, according to the informal approach of the usability tests [15], the same scenarios are presented to all participants to identify the main issues of the site despite the characteristics of the people. Following the completion of the scenario, the evaluator completed, with the user, the Post Study System Usability Questionnaire (PSSUQ) (the free translation of the questionnaire) in the form of an interview.

The purpose of the data analysis was to identify and document the problems and aspects appreciated when using the site and then prioritize problem solving. The data analysis was carried out in three steps:

User Profile: The data collected in relation to the user profile was tabulated to interpret the results and enrich the discussion.

Scenario analysis: A theoretical thematic analysis was conducted using Braun & Clarke [19] approach for the eight sessions (content of video recordings, observation grids and user comments/suggestions) by categorizing the data according to the main themes presented in the PSSUQ using a hypothetical-deductive process [19]. This was done in order to highlight the problems and positive aspects and thus to highlight the trends and singularities related to ease of use specifically.

Analysis of the questionnaires: The responses to the PSSUQ questions were further developed through participant feedback and allowed for a better understanding of the problems and positive aspects identified by users. This data was included in the theoretical thematic analysis.

Results

Two clinicians (Neuropsychologist and Physiotherapist), two students (Student1 and Student2), two stroke survivors (Stroke1 and Stroke2) and two relatives (Niece and Sister), aged 21 to 83, participated in the study (Table 1). All participants had been using a computer for more than 5 years and had visited websites several times a day, with the exception of Stroke1, who mentioned visiting websites several times a week (Table 1). Stroke1, Stroke2 and Niece had never visited the Strokengine site (Table I).

Table 1: Participants’ characteristics (n=8).




  
    Participant 
    Age 
       
    Time    spent on the Internet per week 
    Frequency    of website visits 
    Experience    with Strokengine

      (number of times) 
  
  
    Neuropsychologist 
    48 
    5h    and + 
    Several    x/day 
    Yes    (1-5x) 
  
  
    Physiotherapist 
    30 
    5h    and + 
    Several    x/day 
    Yes    (6-10x) 
  
  
    Student1 
    23 
    5h    and + 
    Several    x/day 
    Yes    (20x +) 
  
  
    Student2 
    23 
    5h    and + 
    Several    x/day 
    Yes    (20x +) 
  
  
    Stroke1 
    83 
    3h    < 5h 
    Several    x/week 
    No 
  
  
    Stroke2 
    61 
    5h    and + 
    Several    x/day 
    No 
  
  
    Niece 
    21 
    5h    and + 
    Several    x/day 
    No 
  
  
    Sister 
    29 
    5h    and + 
    Several    x/day 
    Yes    (1-5x) 
  
  
    x=times



Table 1: Participants’ characteristics (n=8).

All eight participants stated that the site was easy to use according to response frequencies. Only one answer disagree was given by Niece and it was in relation to the quality of the interface. The remaining responses are divided between strongly agree (n=9), agree (n=16), somewhat agree (n=4) and neither agree nor disagree (n=2) (Table 2). One participant (Neuropsychologist) indicated that it is easy to learn to use this site because it is intuitive. However, for Student1 and Niece, the interventions section could be more organized, as it was necessary to go through several paths before finding the description of interventions related to a specific problem (e.g. dysphagia). As a result, according to them, some paths are not necessarily intuitive to find information. However, Neuropsychologist and Student1 indicated that most of the information was found quite quickly (2-3 clicks). Physiotherapist and Student2 appreciated the fact that the different sections of the site are on the home page, but this can also be confusing given the large amount of text. Stroke2 noted that the site is easy to use, but difficult to find. According to him, the terms stroke and engine do not directly refer to stroke rehabilitation. Therefore, he perceived that a person who doesn’t know the site, but who looks for factual information on stroke rehabilitation, by searching on Google, wouldn’t find it, because it’s not one of the first sites that will come out.

Table 2: Answers about the ease of use of Strokengine.




  
    Components defining ease of use 
    Strongly agree 
      (n) 
    Agree 
      (n) 
    Somewhat agree 
      (n) 
    Neither agree nor disagree 
      (n) 
    Rather disagree 
      (n) 
    Disagree 
      (n) 
    Strongly disagree 
      (n) 
  
  
    Overall    user satisfaction 
    1 
    5 
    1 
    1 
    0 
    0 
    0 
  
  
    Usefulness    of the system 
    2 
    5 
    1 
    0 
    0 
    0 
    0 
  
  
    Quality    of information 
    3 
    3 
    2 
    0 
    0 
    0 
    0 
  
  
    Quality    of the interface 
    3 
    3 
    0 
    1 
    0 
    1 
    0



Table 2: Answers about the ease of use of Strokengine.

Ease of use was further described according to four components (based on the PSSUQ): overall satisfaction, usefulness of the site, quality of information and quality of the interface (Table 3). Overall satisfaction was high with the exception of Niece who indicated neutral overall satisfaction, as she perceived the role of the relative not being sufficiently highlighted. Despite areas for improvements, the site was perceived useful as both clinicians stated that they will recommend the site to their patients in the future whereas for people who have had a stroke, the site was perceived as very useful for society and they want to consult it again to answer their questions. Regarding the quality of information, all users stated that they would recommend the use of this site since the information is based on evidence. Sister raised a question about the frequency of updating the information on the site. Participants had many positive comments about the quality of the interface but Neuropsychologist, Stroke2 and Niece mentioned that that a modernization of the interface would be appropriate. The use of icons was highly appreciated by Student1 and Sister, as it makes it easier to find information. However, Niece and Stroke2 were not able to identify the icons in the assessment and intervention sections on the right side of the screen, as they did not look like the icons on the left. In the sections related to the evolution of stroke recovery, at the case story level, for Stroke2 and Sister, having a picture of a woman over one story and a picture of a man over the other is confusing, as it leads us to believe that these are case histories that differ by gender and not by the side of stroke.

Table 3: Examples of verbatim excerpts about the overall ease of use of Strokengine and its components (n=8).

Ease of use
Positive aspects

For a person without cognitive impairments, this site is easy to use (Stroke2).
The information is found fairly quickly, maximum 2-3 clicks (Student1).
It's a good thing that most of the site is on the home page, but it can be confusing. However, it is easily accessible (Physiotherapist).

Areas for improvement

For clinicians, the interventions are not explained, for example, I can't automatically click on acupuncture and it directs me to the explanation page. To access it, you must click on associated interventions on the right of the screen; this is not intuitive (Student2).
If I look at the ADL section in interventions, for example, I don't find the information relevant (Niece).
The name of the site is not intuitive. In the sense that, as a patient, I wouldn't think of looking for a Stroke Engine, but rather a stroke or stroke rehabilitation. So how would I know that this site exists? (Stroke2).

Overall satisfaction
Positive aspects

I will recommend the site to my patients (Neuropsychologist).
I will advocate the use of this site as it is evidence-based (Physiotherapist).

Areas for improvement

If I have to ask for help, I don't know where to go, emailing is too long (Student1).
We should make sure that if I click on French as a language, it doesn't turn into English when I click on another section (Student1).

Usefulness of the system
Positive aspects

For clinicians, it would be relevant if we could access the site's new features on the Facebook page so that we can stay up to date on stroke rehabilitation without having to search the entire site every time (Physiotherapist).
I find the site very pleasant and useful...very useful for society (Stroke1).
It is a beautiful tool that can be complementary to all the information that occupational and physiotherapists give (Sister).

Areas for improvement

The site does not include detailed sections specifically for family members; it is up to me to choose which information is relevant between patient information and clinician information (Niece).

Quality of information
Positive aspects

I will use Strokengine more for assessments and interventions that are well documented; this will allow me to be productive in my schoolwork because everything in the literature is gathered in one place (Student1).

Areas for improvement

The consequences of a stroke are not clearly indicated (Physiotherapist).
More emphasis should be placed on information about what a stroke is by adding, for example, a definition button, rather than searching through the buttons of the evolution of stroke" (Student1).
If I'm looking for information about what a stroke is, I'll go to Wikipedia instead, it's faster (Student2).

Quality of the interface
Positive aspects

There is no information overload and colour contrasts; this is beneficial for people with possible cognitive impairments" (Neuropsychologist).
The fact that the police can be enlarged and all the information is in the middle of the page is adequate for people with hemineglect and/or perceptual difficulties" (Neuropsychologist).
Colored icons help to find information easily (Sister).
The search engine at the top right is relevant (Stroke2).
The site is really very well done (Stroke1).

Areas for improvement

Too many font differences between titles and descriptions" (Student1).
There's too much little text that doesn't make you want to read the rest. The clearest sections are those of assessments and interventions (Student1).
If the person has cognitive difficulties or low concentration, having a lot of text can be difficult (Sister).
There is a lack of organization at the site level (Student2).
Aesthetically, I think the site is past date; the design is more early 2000 (Niece).



Table 3: Examples of verbatim excerpts about the overall ease of use of Strokengine and its components (n=8).

Discussion and Conclusion

Discussion

The purpose of this study was to assess and describe the perceived ease of use of Strokengine by site users so that it could eventually be improved by optimizing site navigation and meeting users’ needs. The first study of the Stroke Engine-Family platform and its ease of use was conducted more than 10 years ago with individuals who have had a stroke and their families [9]. For the 2012 platform, called the Strokengine (a modified version of the 2009 platform), ease of use was evaluated with clinicians [8]. The present study assessed the ease of use of the 2018 version of the site by clinicians, students, stroke survivors and families of stroke survivors. As a result, this gives a more complete picture of the users’ perception of ease of use, as it has been evaluated with the full target audience.

In line with the two previous studies [8,9], the results of this study indicated that the site is easy to use given the high frequency of positive responses and comments. Participants made several recommendations to improve the site and make it even easier to use. It would be useful to facilitate direct access to this site from Google or any other search engine. Indeed, for someone who is looking for information on stroke rehabilitation and does not know that this site exists, the terms stroke and engine are not necessarily intuitive. However, despite this, according to Google Analytics, the website receives more than 8 000 visitors weekly which makes it the go to site for evidence-based information on stroke rehabilitation. To increase user satisfaction with the site, it would be relevant to highlight better the role of the family member. For example, a specific tab entitled Information for Relatives could be added or a capsule, per intervention, that summarizes what the family member can do. Participants also recommended using another means of contacting the team besides email. However, this recommendation was made on bad experience with other website as strokengine team answers email within two working days. Often, users prefer Facebook instead of an email address because it is perceived as a faster way to communicate. Another recommendation would be to update the Facebook page of the site so that users can quickly access new features and to expand the button that allows access to the Facebook page from the site accordingly. Social networks are growing in popularity and it is interesting to use them to give more visibility to the site, as they help to increase the number of new visitors while maintaining the number of recurring visitors [20]. As such, a twitter account, using the words stroke and rehabilitation (https://twitter.com/StrokeRehabili1), was created and the Facebook page was updated (www.facebook.com/ strokengine.infoavc). Both are now used to inform users of additions and updates to the website.

In relation to the quality of information, it would be appropriate to further organize information on what a stroke is and its impacts by adding a specific tab to this on the home page. Finally, in relation to the quality of the interface, it would be appropriate to reduce the font differences between titles and descriptions according to each section to standardize everything. By integrating a table of contents on the home page, this would allow visitors to see what the site contains and would facilitate navigation on the site. Since two participants had difficulty locating the icons in the assessment and intervention sections on the right-hand side of the screen, the organization of the home page should be reviewed with a focus on the important sections. In the sections related to the evolution of stroke recovery, at the case history level, it would be appropriate to remove the pictures over the case histories and indicate only the symptom side as the title to decrease potential confusion with gender. Finally, the bright colours were appreciated, but a modernization of the interface would be appropriate. Indeed, studies show that regular modernization of the interface is essential to promote a pleasant user experience [21] and actual interface is from 2015.

This study used the methodological approaches for the evaluation of health information systems [16] to build its own tests, described and applied a rigorous methodology and detailed process that could be used to evaluate any online resource, whether health-related or not. The use of various forms of data collection including videos, verbal feedback, written responses to Likert-type questions and observations of interactions with the system in real time were extremely useful for this study, as we were able to generate a large amount of data related to the themes. Although there have been few studies on the ease of use of health websites, another study has also promoted this type of data collection [22], as it allows information to be triangulated according to each participant and between participants, thus providing the most comprehensive picture related to ease of use [23]. Hinchliffe and Mummery [22] added another component to their study, namely to make the site modifications that emerged during the usability assessment. Then, they conducted a second usability assessment to determine if the site is easier to use as a result of the changes. In another study related to Strokengine, it would be interesting to follow the same method and make the changes raised in this study and then reassess the ease of use. One possible limitation of this study is the small sample size. However, with 8 participants, the majority of problems should be detected [14]. Also, given the methods used, we were able to have as many comments and suggestions as possible. Participants were also chosen for the variability of their characteristics, whether it be the different profiles (e.g. clinician or student), living in different countries (Canada and Belgium) and age (ranging from 21 to 83 years) in order to have the most global picture possible. Another possible limitation of this study is that not all sections of the site were consulted. However, the scenario was designed to ensure that participants consulted the main sections of the site that may have an impact on ease of use.

Conclusion

This study allowed us to obtain comments and suggestions from the main users of the site regarding the ease of use of Strokengine to eventually optimize navigation and maximize its potential. In Canada, the population has a low level of health literacy as studies have shown that 60% of adults and 88% of seniors lack the skills to find, understand and use information in order to make healthy decisions [24]. Given that in 2018, 91% of the Canadian population was using the Internet [25], the use of online resources, such as Strokengine, that popularize health information to make it more accessible should be promoted by clinicians to help patients and their families make better informed decisions for their health.

Funding

This work was supported by the Canadian Partnership for Stroke Recovery. AR is supported by a senior career award from the Quebec Research Agency – Heath (FRQ-S).

References

Download PDF

Citation: Chibane Y and Rochette A. Qualitative Study on the Perceived Ease of Use of a Knowledge Translation Website: www.strokengine.ca. Phys Med Rehabil Int. 2020; 7(1): 1168.

Home

Journal Scope

Editorial Board

Instruction for Authors

Submit Your Article