Methodological Considerations in the Longitudinal Analysis of Cognition in Older Adults

Editorial

Longitudinal assessments of cognition are commonplace in studies of older adults, particular as they relate to the development of clinically significant cognitive problems, such as Mild Cognitive Impairment (MCI) and Alzheimer’s disease (AD). However, the interest in this area also lies in characterizing age-related normative changes in cognition and how these changes correlate with physical function, quality of life, and other measures of self-reported wellbeing. Although there is a wealth of literature describing longitudinal changes in cognition among older adults, there are some issues that must be taken into consideration when reading and interpreting the findings of these studies.

One of the most important, but oft ignored issues, is that of practice effects which refers to an improvement in future test performance based on prior exposure [1,2]. This issue is of particular importance when cognitive tests are being used to determine whether or not an individual is developing MCI or AD [3] as clinically significant change may be obscured by practice effects [4]. When investigating age-related changes in cognition and their association with noncognitive measures such as quality of life, physical activity level, and functional status, practice effects may lead to an overestimate of true cognitive performance over time. This is because individuals may have “learned” the test and may have memorized certain portions of the test where the content remains the same at each assessment (e.g., list of words used for memory recall). The use of equivalent alternate forms for cognitive tests is one way to help minimize practice effects. Tests such as the Montreal Cognitive Assessment (MoCA), Rey Auditory Verbal Learning Test (RAVLT), and the Mattis Dementia Rating Scale 2 (DRS-2) offer equivalent alternate forms in order to minimize practice effects in serial administrations as they contain different versions of the same test items.

In addition to the impact that practice effects may have on observed changes in cognition, within-subject variability is another significant problem that often confounds longitudinal assessment of cognition. Even among individuals who remain cognitively normal over a long period of time, significant fluctuations in performance between assessments is quite common [5]. Much of this has to do with the natural variation of cognitive performance that is inherent in all individuals, but this variation may also be impacted by changes in co morbid medical conditions, medication use, mood and affect, and other idiosyncratic intrapersonal factors [6,7]. These within- subject factors that impact test performance manifest themselves empirically via large standard deviations for the rate of change across time points. In some cases, the variability of test performance may be numerically similar to the average rate of change for a cognitive test [8]. From a statistical standpoint, this brings significant challenges since cognitive test effect sizes are a function of the ratio between the mean and standard deviation. Thus, larger standard deviations result in smaller effect sizes which may lead to non-significant group differences or weak associations with other continuous variables.

Others have pointed out that the effect sizes for cognitive outcomes are inherently small, which results in the need for larger sample sizes in clinical trials [8]. Among MCI and AD trials, this issue has been particularly problematic and the lack of significant differences between placebo and treatment groups in several large AD clinical trials has played a role in these studies failing to show efficacy [9]. Since the underlying pathological changes associated with AD are thought to occur several years before the onset of clinical symptoms [10], it is possible that the degree of pathology present at the time when clinical symptoms are manifest is simply too great for any compound to have a meaningful effect [11]. This has led to the initiation of prevention trials for AD in which asymptomatic individuals at risk for developing AD are enrolled with hope of delaying or preventing the onset of clinical AD [12]. Although this represents a major shift in the paradigm of AD clinical research, one of the main issues that researchers and clinical trialists must grapple with is the need to detect significant treatment differences among individuals who are cognitively normal. Two on-going prevention studies have developed and validated composite test scores that utilize several different neuropsychological tests that have shown to be sensitive to the cognitive changes associated with pre-clinical AD [13,14]. However, it has been suggested that the sensitivity of these composite cognitive tests may be enhanced when coupled with biomarkers of disease progression [15].

A possible solution to the problem of detecting significant group differences on cognitive measures would be to utilize the Reliable Change Index (RCI) which is a subject-based method of determining clinically significant change between assessments [16]. Using an individual-based metric of change offers the advantage of accounting for the within-subject variability that can often add extraneous variability to a dataset. The RCI corrects for practice effects and instrument reliability and is then able to quantify the degree of change (e.g., number of points on a test) necessary to state whether an individual has shown a clinically significant change from one assessment to the next. The RCI can be positive or negative, which allows a clinician or researcher to determine whether clinically significant improvement or decline has occurred for an individual. The RCI could be used in clinical trials to create a binary outcome indicating whether clinically significant change occurred. Specifically, the proportion of individuals who show clinically significant change in the treatment and placebo groups could be compared and then be used as a measure of drug efficacy. Using the RCI to create a binary outcome would then allow for other measures of clinical significance, such as the Number Needed to Treat (NNT), to be also be used in order to determine what impact a new drug may have at the population level.

Observational studies and clinical trials that assess longitudinal changes in cognition are both subject to the negative impact that practice effects and intraindividual variability have on the ability to detect significant group differences or treatment effects. An additional challenge is that effect sizes for group/treatment differences are inherently small which underscores the importance of minimizing sources of extraneous variability. These issues are of particular importance to the field of AD treatment and research given the field’s shift toward to the conduct of prevention studies where the detection of significant, but subtle changes in cognition is of utmost importance. Observational studies are also subject to the detrimental impacts of practice effects and intraindividual variability as they relate to accurately assessing the natural course of age-associated changes in cognition and their associations other psychosocial and functional constructs.

Although accurately assessing longitudinal change in cognition is subject to a number of confounding factors, the impact of these factors may be mitigated through methodologic strategies (e.g., alternating test forms) and also through statistical procedures that account for practice effects and measurement error (e.g., RCI). These procedures should be implemented when possible and can be utilized in both intervention and observational studies. In doing so, intraindividual cognitive changes over time will be better characterized and less susceptible to the detrimental impact of practice effects and measurement error.

References

1. Bartels C, Wegrzyn M, Wiedl A, Ackermann V, Ehrenreich H. Practice Effects in Healthy Adults: A Longitudinal Study on Frequent Repetitive Cognitive Testing. BMC Neuroscience. 2010; 11: 118.
2. Salthouse TA. Influence of age on practice effects in longitudinal neurocognitive change. Neuropsychology. 2010; 24: 563-572.
3. Duff K, Beglinger LJ, Van Der Heiden S, Moser DJ, Arndt S, Schultz SK, et al. Short-term practice effects in amnestic mild cognitive impairment: implications for diagnosis and treatment. Int Psychogeriatr. 2008; 20: 986-999.
4. Salthouse TA. Frequent Cognitive Assessments May Obscure Cognitive Decline. Psychol Assess. 2014; 26: 1063-1069.
5. Sliwinski M, Buschke H. Modeling Intraindividual Cognitive Change in Aging Adults: Results from the Einstein Aging Studies. Neuropsychol Dev Cogn B Aging NeuropsycholCogn. 2004; 11: 196-211.
6. Piccinin AM, Muniz G, Sparks C, Bontempo DE. An evaluation of analytical approaches for understanding change in cognition in the context of aging and health. J Gerontol B Psychol Sci Soc Sci. 2011; 66: 36-49.
7. Salthouse TA. Correlates of cognitive change. J Exp Psychol Gen. 2014; 143: 1026-1048.
8. Knopman DS. Clinical trial design issues in mild to moderate Alzheimer disease. Cogn Behav Neurol. 2008; 21: 197-201.
9. Becker RE, Greig NH, Giacobini E. Why do so many drugs for Alzheimer's disease fail in development? Time for new methods and new practices? J Alzheimers Dis. 2008; 15: 303-325.
10. Jack CR Jr, Knopman DS, Jagust WJ, Shaw LM, Aisen PS, Weiner MW, et al. Hypothetical model of dynamic biomarkers of the Alzheimer's pathological cascade. Lancet Neurol. 2010; 9: 119-128.
11. Kozauer N, Katz R. Regulatory innovation and drug development for early-stage Alzheimer's disease. N Engl J Med. 2013; 368: 1169-1171.
12. Reiman EM, Langbaum JBS, Tariot PN. Alzheimer’s Prevention Initiative: A Proposal to Evaluate Presymptomatic Treatments as Quickly as Possible. Biomark Med. 2010; 4: 3-14.
13. Langbaum JB, Hendrix SB, Ayutyanont N, Chen K, Fleisher AS, Shah RC et al. An Empirically Derived Composite Cognitive Test Score With Improved Power to Track and Evaluate Treatments for Preclinical Alzheimer’s Disease. Alzheimers Dement. 2014; 10: 666-674.
14. Sperling RA, Rentz DM, Johnson KA, Karlawish J, Donohue M, Salmon DP, et al. The A4 study: stopping AD before symptoms begin? Sci Transl Med. 2014; 6: 228fs 13.
15. Hendrix SB. Measuring clinical progression in MCI and pre-MCI populations: enrichment and optimizing clinical outcomes over time. Alzheimers Res Ther. 2012; 4: 24.
16. Duff K. Evidence-Based Indicators of Neuropsychological Change in the Individual Patient: Relevant Concepts and Methods. Arch ClinNeuropsychol. 2012; 27: 248-261.