Special Article - Pediatric Rehabilitation
Phys Med Rehabil Int. 2015; 2(9): 1069.
Reliability of Movakic; An Instrument to Evaluate Motor Abilities in Children with Severe Multiple Disabilities (Part-II)
Mensch SM1,2,5*, Rameckers EAA3,4,5, Echteld MA1,6,7 and Evenhuis HM¹
¹Intellectual Disability Medicine, Department of General Practice, Erasmus Medical Centre, The Netherlands
²Ipse de Bruggen, Centre of Expertise in Intellectual Disabilities, The Netherlands
³Department of Rehabilitation Medicine, School for Public Health and Primary care (CAPHRI), Maastricht University, The Netherlands
4Adelante Centre of Expertise in Rehabilitation and Audiology, The Netherlands
5University for Professionals for Paediatric Physical Therapy, AVANSplus, The Netherlands
6Stichting Wetenschap Balans, The Netherlands
7Prisma, Centre of Expertise in Intellectual Disabilities, The Netherlands
*Corresponding author: Sonja M Mensch, Erasmus Medical Centre, Intellectual Disability Medicine, Department of General Practice, P.O. Box 2040, 3000 CA Rotterdam, The Netherlands
Received: October 16, 2015; Accepted: November 17, 2015; Published: November 20, 2015
Abstract
Aim: ‘MOtor eVAluation of KIds with multiple and Complex disabilities’ (Movakic) is a newly developed Dutch instrument for evaluating motor abilities in children with severe multiple disabilities. We have previously shown that its feasibility and content validity are satisfactory. The aim of this study is to investigate test-retest and inter-rater reliability.
Methods: Children with severe multiple disabilities were scored six times by their own physical therapists at a three month interval, and at baseline by a second therapist familiar with the child in a subset of children. For the purposes of this study, the three-month period in which no event involving the child took place was selected.
Results: Sixty children were recruited. The mean age of the children was 7.7 years (range 2-16), 45% had a cognitive development level <6 months (N=27) and 52% had Gross Motor Function Classification System level V (N=31). Test-retest reliability could be evaluated in 50 children and inter-rater reliability in 19 children. Intraclass correlations were all excellent or good (range .72-.98). Adequate absolute reliability is reflected in a small mean distance of Movakic scores and most respondents’ distances for test and retest were between one standard deviation and zero. Distribution is not related to the score level, although a ceiling effect might be present in score range 90-100.
Conclusion: Movakic is a reliable instrument for measuring motor abilities in children with severe multiple disabilities.
Keywords: Severe multiple disabilities; Motor abilities; Cerebral Palsy GMFCS V; Evaluative instrument; Movakic; Reliability
Abbreviations
Movakic: MOtor eVAluation of KIds with multiple and Complex disabilities; SMD: Severe Multiple Disabilities; GMFCS: Gross Motor Function Classification System; COSMIN: COnsensus-based Standards for the selection of health Measurement Instruments; IQ: Intelligent Quotient; ICF-CY: International Classification of Functioning, Disability and Health for Children and Youth; ICC: intraclass correlation coefficients.
Introduction
Motor abilities are of paramount importance to independent functioning, but are often severely compromised in children with severe multiple disabilities (SMD). Children with SMD suffer from profound intellectual disabilities (IQ <25) and have a level of motor abilities that is comparable to level IV/V on the Gross Motor Function Classification System (GMFCS) for children with cerebral palsy [1,2]. In addition, children with SMD may have multiple sensory disorders and other comorbidities. Usually, physiotherapists are closely involved with stimulation and training of such children, because even subtle improvement of motor abilities can aid these children in developing some degree of control over their environment and may as such improve their quality of life. As in any healthcare profession, physical therapists desire to evaluate the effectiveness of their treatment methods, for which reliable instruments are needed.
However, commonly used instruments for measuring motor abilities in children with disabilities [3] are considered unsuitable for children with SMD. They are unsuitable because of their inclusion of higher GMFCS levels, the requirement of perfect execution of the motor ability, the need for verbal instruction, the use of large-step grading, and the design of items without the application of manual support or use of devices in mind. Therefore, an instrument was needed that fulfills specific suitability criteria [4].
A new instrument named Movakic (MOtor eVAluation of KIds with multiple and Complex disabilities) for measuring and evaluating motor abilities in children with SMD was developed by a Dutch expert focus group and was found to be feasible with good content validity [4]. Before an instrument can be used in clinical or research settings, stability across time and raters should be assessed [5]. Therefore, in this study the test-retest reliability and inter-rater reliability of Movakic were evaluated.
Methods
Participants, selection
Children younger than 18 years with SMD who received care in specialized day-care centers were included. Severe multiple disabilities were defined as profound intellectual disability (IQ<25) in combination with severely impaired motor abilities GMFCS level IV and V [1].
Each of 37 experienced therapists working in the centers, who all had more than 10 years experience with the target population, selected one or two of their own clients. Informed consent by parents or legal representatives was obtained from all participants.
Movakic
Movakic is a questionnaire consisting of items on motor abilities; the complete questionnaire is shown in the appendix of the design article [4]. All terminology used is based on the International Classification of Functioning, Disability and Health for Children and Youth (ICF-CY) terminology [6]. Motor abilities are distributed over 13 ‘situations’ (Table 1), each representing a client’s body position with or without the use of a device. Within each situation, a cluster of items addresses four groups of motor abilities: maintaining position, activities, changing body position, and moving around. Questions are asked about the extent in which manual support or support from a device was needed, the own activity of the child, and the extent of aiding the child manually i.e. ‘provocation’ of the child with your hands.
Positions
Lying
Sitting
Standing
Situations (13) →
Grouping of motor abilities
↓
1 Supine
2 Supine with device
3 Prone
4 Prone with device
5 Side
6 Side with device
7 Flat surface
8 Dangling legs
9 Chair/ sitting device
10 feet on subsurface
11 Without device
12 With device
13 Care situation
Maintaining position
Items with questions on
1 Extent of manual or support by device
2 Activity of the child
3 Extent of manual facilitation/ stimulation
Activities
Changing body position
Moving around
Table 1: Structure of Movakic.
Movakic has to be completed by professional therapists who are familiar with the clients’ motor performance on the basis of longer treatment experience with the child, and not on performance of activities in a specific test situation, which is the basis of most other observational instruments. Scoring of motor abilities in this group should not depend on functioning in a single test situation only, because performance may worsen under the influence of lack of attention, fatigue, bad health, medication use, or unfamiliar circumstances [4]. Because all children have different abilities or disabilities and different therapeutic goals, only the questions regarding situations that are relevant to the child need to be scored: A situation could be relevant if a baseline measurement is desired for future follow-up, if therapeutic changes are expected in a certain situation, or for evaluating change in motor ability.
All items are scored on a five-point Likert-scale. Scores on the left-hand side always represent lower and scores on the right-hand side represent higher scores in motor function (range 0-4 points). An example of an item is given in Table 2.
1-Maintaining sitting
What is the extent of manual support you gave the child to maintain this position?
Full
None
What is the child’s level of activity?
Full Passive
Full Active
What is your the extent of facilitation to stimulate the motor ability?
Full
None
Table 2: Example of an item including the sub-questions.
The maximum scores of the 13 situations will differ because of the variable cluster of items. Therefore, per situation, absolute scores are converted into percentages (situation score divided by maximum situation score x 100). For the remainder of this paper, this percentage will be referred to as the ‘Movakic score’.
Movakic is provided on-line through a secure Internet site. The completion of the instrument on the screen starts by the items and their questions for the chosen situation. Per question, only one answer (one button) can be selected. Modifications are allowed until the “save” button is pressed. During the study period, participating physiotherapists could not check the results of the Movakic score after completion of the questionnaire. It was only after the study ended and all data was collected that they received a report with the scores for each child.
Procedure
The reliability study is part of a longitudinal study for which participating therapists were instructed in the use of the instrument and the study procedure during a one-day training session on the application and the user manual. Data collection was performed from August 2010 to October 2011. Therapists were requested to complete Movakic six times (T0-T5) in a period of 18 months. Intervals of three months were chosen based on standardized evaluation periods in clinical practice and changes in this period are not expected in the target population if no specific events occur. In order to evaluate inter-rater reliability, a second therapist who was also familiar with the client’s motor performance, if available, completed Movakic at baseline.
For this reliability study, therapists were requested to choose a situation per child, containing a cluster of the items of Movakic. Therapists were asked to note events that might have influenced the child’s scores during each interval. Because of their frequent contacts with the child, its parents, and its cares and other therapists in the daycare setting, physiotherapists were well aware of medical conditions, medication changes, changes in the carer teams or at home, and other events. For the current reliability study, scores between two measurement periods in which no events took place were chosen, so called T0-T1 (Figure 1).
Figure 1: Structure of Movakic.
We recorded the child’s sex, age, estimated cognitive developmental age, GMFCS level, diagnosis, comorbid conditions, and devices, which we asked the child’s therapist to provide.
Analyses
For the assessment of test-retest reliability, baseline (T0) Movakic scores were compared with T1-scores at three months. In case an event was recorded that might have caused a change between T0 and T1, another 3-month period was selected in which no such events had occurred. For the assessment of inter-rater reliability, baseline measurements of two therapists were used.
To test relative reliability, which is the degree in which children maintain their position in a sample over repeated measurements, two-way mixed model intraclass correlation coefficients (ICC) were calculated with 95%-confidence intervals between the Movakic scores of T0 en T1 (for test-retest reliability) and between the Movakic scores of the two raters (for inter-rater reliability). ICC’s for test-retest and inter-rater reliability were also calculated for each of the four groups of motor abilities. Reliability was classified as excellent (> 0.75), good (0.60–0.74), fair (0.40-0.59), or poor (<0.40) [7].
The ICC does not provide information about the degree in which actual scores for an individual vary over repeated measurements (absolute reliability). The smaller the differences, the higher the absolute reliability. This was done by performing Bland & Altman analyses [8]. First, we calculated for each child the distance (absolute difference) between T0 and T1 Movakic scores (for test-retest reliability) and between both T0 Movakic scores per rater (for inter-rater reliability). Next, for test-retest reliability mean Movakic scores at T0 and T1 were plotted against the individual differences between T0 and T1, for inter-rater reliability mean Movakic scores were plotted against individual differences between both raters at T0. Adequate reliability is represented by small differences from the mean (within one standard deviation (SD) of the mean).
All calculations were performed in IBM Statistical Package for the Social Sciences version 21.
Results
Population
Sixty children with a mean age of 7.7 years (range 2-16 years) were selected in 15 different day-care centers. Characteristics of the study population are listed in Table 3.
N = 60
%
Gender
Male
Female
31
29
52
48
Age in years
1-6
6-12
12-18
20
28
12
33
47
20
Cognitive development level in months
0-6
6-12
12-18
>18
Not scored
27
12
5
2
14
45
20
8
3
23
GMFCS*1 level
IV
V
Not scored
22
31
7
37
52
11
Diagnosis
Cerebral Palsy
Spastic CP
Ataxic CP
Dyskinetic CP
Syndromes/ gene mutations
Acquired brain injury
Metabolic disease
Unknown
25
21
1
3
18
1
2
14
42
84
4
12
30
1.7
3
23
Comorbidity
Epilepsy
PEG*2 tube
Scoliosis
Visual impairment
Respiratory problem
Other*3
Secondary problem/ contractures
39
22
24
44
17
17
29
65
37
40
73
28
28
48
Devices*4
Wheelchair
Standing device
Walking aid
(Semi) Orthopaedic shoes
Orthotics
Lying device
57
42
24
24
32
16
95
70
40
40
53
27
*1GMFCS: Gross Motor Function Classification System [1].
*2PEG tube: percutaneous endoscopic gastrostomy tube.
*3Other: additionally noted comorbid conditions such as heart disease, hearing disorders, diabetes mellitus, gastro-oesophageal reflux disease.
*4Device: Assistive devices and aids for personal mobility [6].
Table 3: Characteristics of the study population.
Test-retest reliability
During the interval of three months, six children were lost to follow-up: one died, two were severely ill, two had been transferred to other day-care centers and one moved home. In addition, four children were lost to follow-up for test-retest analyses due to noted events that might have influenced reliability during all of the five 3-month intervals, such as surgery, sickness or increasing contractures. Test-retest reliability for the remaining 50 children in terms of agreement (ICC) and mean distance of Movakic scores is presented in Table 4.
Baseline – 3 months
ICC (95%CI)
Mean distance of Movakic scores (SD)
Total score
.95** (.92-.97)
6.8 (6.4)
Group of motor ability
Maintaining position
.98** (.97-.99)
Activities
.96** (.94-.98)
Changing body position
.91** (.82-.95)
Moving around
.72** (.27-.90)
ICC: intraclass correlation coefficient; 95%CI: 95% Confidence Interval; SD: Standard Deviation.
*p <0.05, **p<0.01.
Table 4: Test-retest reliability (N=50).
Figure 2 shows the Bland & Altman plot of all individual differences between Movakic scores at T0 and T1 scores against mean Movakic scores. The score differences varied between -25 and 24 points (SD = 9.2). Seventy-four percent of the differences were within one standard deviation (dotted lines).
Figure 2: Individual differences of Movakic score between T0 and T1
(y-axis), plotted against mean Movakic score (x-axis) (N=50). The dotted
lines represent one standard deviation of the differences in Movakic scores.
Inter-rater reliability
Movakic was completed at baseline by two therapists for 19 children. Inter-rater reliability in terms of ICC and mean distance of Movakic total scores is presented in Table 5.
Baseline
ICC (95%CI)
Mean distance of Movakic scores (SD)
Total score
.94** (.85-.98)
8.6 (8.7)
Group of motor ability
Maintaining position
.89** (.35-.97)
Activities
.96** (.89-.98)
Changing body position
.85** (.57-.95)
Moving around
.97** (.90-.99)
ICC: intraclass correlation coefficient; 95%CI: 95% Confidence Interval; SD: Standard Deviation.
*p <0.05, **p<0.01.
Table 5: Inter-rater reliability (N=19).
Figure 3 shows the Bland & Altman plot of all individual differences between Movakic scores of the two raters against mean Movakic scores. The score distances varied between -21 and 29 points (SD=11.8). Seventy-nine percent of the distances were within the standard deviation (dotted lines in the figure).
Figure 3: Individual differences in Movakic scores between two raters
(y-axis), plotted against mean Movakic scores (x-axis) (N=19). The dotted
lines represent one standard deviation of the differences.
Discussion
All ICC values for test-retest reliability were excellent, except for the subscale ‘Moving around’, of which the ICC was good. Here, the 95% confidence interval of ICC’s was very wide, with the lowest limit at .27. The 95% confidence interval for ‘Changing body position’ was also relatively wide compared to the other ICCs. ICC values for interrater reliability of the subscales ‘Maintaining position’ and ‘Changing body position’ were excellent, but the 95% confidence interval of the ICC’s was wide, with the lowest values at .57 and at .35, respectively. Adequate absolute reliability is reflected in a small difference of mean Movakic scores, for test-retest reliability 6.8 (SD 6.4) and for interrater reliability 8.6 (SD 8.7). Adequate reliability is also reflected in 74% and 79% of the test-retest and inter-rater distances falling within one SD from zero. However, a few respondents exhibited large distance scores, indicating suboptimal reliability for a small number of respondents making some caution in interpreting the results.
In fact, we expected lower ICC’s because, in accordance with the procedure, the therapists selected a relevant situation for each individual child. Their choice depended on various aspects such as diagnosis, comorbidity, therapeutic goals, used devices and abilities and disabilities. This procedure has ecological validity, i.e. represents the real-life situation, but also creates diversity, which may be reflected in the suboptimal ICC’s with wide confidence intervals. It was hard to find a more standardized procedure for this heterogeneous and specific target group. The heterogeneity of the study population is apparent from Table 1. The children were recruited from wide spread care organizations in the country so we can assume the study population is representative and adequately reflects the diversity of the target group. It is encouraging that reliability levels were adequate despite this heterogeneity.
A lower concordance of inter-rater than of test-retest scores was to be expected, because of the relative small number of participants, but also because of the subjectivity introduced by therapists having to estimate the extent in which they used manual support. Nevertheless, the high ICC’s (Tables 2 and 3) support the strong design of Movakic, including the application of manual support.
This study has several strengths. It is one of the few studies in this specific target population with relatively high participation rates e.g., [2,9-16]. Moreover, all participating therapists received adequate training in the use of the instrument related to the purpose of the study. Additionally, all had extensive experience with the target population and were familiar with the included children. The therapists thus fulfilled the prerequisites for participating in the study as raters. Electronic data collection ensured that the data is of good quality and scoring errors were limited. A memory effect was highly minimalized by not showing scores during completion of the questionnaire and the wide time frame of three months. Moreover, therapists noted events that may have influenced the children’s scores during the interval and such intervals were not used for analyses.
The time frame of three months may be considered a long period for showing stability of an instrument. However, based on the experience in clinical care of this group of children, it was expected that stability in motor abilities would be high.
On the other hand, the high ICC’s may be explained by the extreme familiarity of the therapists with the children, the fact that they all had a long-time experience with the target group, but also by the fact that they were highly motivated because of being personally involved in the development of a new instrument. This situation may be less favorable in future daily practice, where physiotherapists with less experience with these children may have to complete Movakic, who will not always be connected to specialized day-care centers, whose time is not or to a limited extent reimbursed by health insurances, or who work in countries with different healthcare systems. Therefore, training is needed to enhance a correct application of the instrument; even in less favorable circumstances [4].
Conclusion
In this study, test-retest and inter-rater reliability were examined of a newly developed instrument, Movakic, to evaluate motor abilities of the specific subgroup of children with SMD [4]. Test-retest reliability was evaluated in 50 children and inter-rater reliability in 19 children with SMD. For both test-retest and inter-rater reliability, intraclass correlations of Movakic scores and of sub-scores for four groups of motor abilities were all excellent or good. Adequate absolute reliability was reflected in a small mean distance of Movakic scores and in the accuracy of individual scores, representing a normal variation.
In addition, from this study we can conclude Movakic is a reliable instrument for measuring motor abilities in children with SMD. But before the implementation of Movakic for clinical evaluation of motor abilities in children with SMD, its responsiveness and construct validity have to be evaluated first.
Acknowledgements
Ipse de Bruggen, care organization for people with intellectual disabilities and Nuts Ohra Fund (nr 1102-045) financed this research. The authors are grateful to the participating therapists for their accurate and convincing collaboration. These participating therapists were from different care organizations all over the Netherlands; Stichting Omega, ‘s Heeren Loo, ASVZ, Zozijn, Stichting Tragel, NOVO, SWZ, Esdégé Reigersdaal, Kempenhaeghe, Kleur (Dichterbij), Schutse, ZMLK school de Kom, Gemiva-SVG, De Hartekamp groep, stichting de Driestroom and Ipse de Bruggen.
References
- Palisano R, Rosenbaum P, Walter S, Russell D, Wood E, Galuppi B. Development and reliability of a system to classify gross motor function in children with cerebral palsy. Dev Med Child Neurol. 1997; 39: 214-223.
- Veugelers R, Calis EA, Penning C, Verhagen A, Bernsen R, Bouquet J, et al. A population-based nested case control study on recurrent pneumonias in children with severe generalized cerebral palsy: Ethical considerations of the design and representativeness of the study sample. BMC Pediatrics. 2005; 5: 25.
- Mensch SM, Rameckers EA, Echteld MA, Evenhuis HM. Instruments for the evaluation of motor abilities for children with severe multiple disabilities: A systematic review of the literature. Res Dev Disabil. 2015; 47: 185-198.
- Mensch SM, Rameckers EEA, Echteld MA, Penning C, Evenhuis HE. Design and content validity of a new instrument to evaluate motor abilities of children with severe multiple disabilities: Movakic. Physical Medicine and Rehabilitation – International. 2015.
- van Kuijk AA, Kosters R, Vugts M, Geurts AC. Treatment for idiopathic toe walking: a systematic review of the literature. J Rehabil Med. 2014; 46: 945-957.
- Cicchetti DV, Sparrow SA. Developing criteria for establishing interrater reliability of specific items: applications to assessment of adaptive behavior. American Journal of Mental Deficiency. 1981; 86: 127-137.
- Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986; 1: 307-310.
- Calis EA, Veugelers R, Sheppard JJ, Tibboel D, Evenhuis HM, Penning C. Dysphagia in children with severe generalized cerebral palsy and intellectual disability. Dev Med Child Neurol. 2008; 50: 625-630.
- Veugelers R, Benninga MA, Calis EA, Willemsen SP, Evenhuis H, Tibboel D, et al. Prevalence and clinical presentation of constipation in children with severe generalized cerebral palsy. Developmental Medicine & Child Neurology. 2010; 52: e216-221.
- Liptak GS, O'Donnell M, Conaway M, Chumlea WC, Wolrey G, Henderson RC, et al. Health status of children with moderate to severe cerebral palsy. Dev Med Child Neurol. 2001; 43: 364-370.
- ten Brug A, van der Putten A, Penne A, Maes B, Vlaskamp C. Multi-sensory storytelling for persons with profound intellectual and multiple disabilities: an analysis of the development, content and application in practice. J Appl Res Intellect Disabil. 2012; 25: 350-359.
- van der Putten A, Vlaskamp C, Reynders K, Nakken H. Movement skill assessment in children with profound multiple disabilities: a psychometric analysis of the top down motor milestone test. Clin Rehabil. 2005; 19: 635-643.
- van der Putten A, Vlaskamp C. Pain assessment in people with profound intellectual and multiple disabilities; a pilot study into the use of the Pain Behaviour Checklist in everyday practice. Res Dev Disabil. 2011; 32: 1677-1684.
- Calis EA, Veugelers R, Rieken R, Tibboel D, Evenhuis HM, Penning C. Energy intake does not correlate with nutritional state in children with severe generalized cerebral palsy and intellectual disability. Clin Nutr. 2010; 29: 617-621.
- Rieken R, van Goudoever JB, Schierbeek H, Willemsen SP, Calis EA, Tibboel D, et al. Measuring body composition and energy expenditure in children with severe neurologic impairment and intellectual disability. Am J Clin Nutr. 2011; 94: 759-766.
Download PDF