The DRAM 3(5) - Addiction and Technology − Reliability of Online Alcohol Use Assessments

Compared to traditional paper and pencil methods, online studies can benefit from the convenience, lower cost, reduced time to completion, access to larger samples, and reduced self-presentation biases offered by the Internet (Miller, 1997; Schmidt, 1997). Reducing the extent that participants bias their self-presentation can benefit addiction-related research. Until now, researchers have not studied the reliability or validity of commonly used addiction-related surveys administered via the Internet. This week’s DRAM reviews a study that, in part, compares the reliability of web-based addiction-related surveys to traditional paper and pencil surveys.

Miller, E.T., Neal, D.J., Roberts, L.J., Baer, J.S., Cressler, S.O., Metrik, J., Marlatt, G.A. (2002) administered a survey to a sample of 255 undergraduates who were assigned randomly to one of three experimental conditions: pencil and paper (PP), web-based (W), and web-based with interruption (WI). The PP and W groups completed the questionnaire without stopping, but the WI group stopped for 1 to 48 hours before finishing the survey. All groups completed the same questionnaire again one week later under the same experimental conditions. In addition to basic demographic information and drinking patterns, the survey included the Alcohol Use Disorders Identification Test (AUDIT;Babor, De la Fluente, Saunders, & Grant, 1992), the Alcohol Dependence Scale (ADS: Skinner & Allen, 1982), the University of Rhode Island Change Assessment (URICA; McConnaughy, DiClemente, Prochaska, & Velicer, 1993) and the Rutgers Alcohol Problem Index (RAPI; White & Labouvie, 1989).

Table 1 – Means scores at time 1, correlation with score at time 2, with significant differences across condition.

The groups did not differ on demographic variables, including age, gender, ethnicity, residence, and also did not differ on measures of alcohol use. Using correlations to measure agreement between first and second administration of the test instruments, the researchers found for all groups that all Pearson correlations were statistically significant (p < .01) and ranged from .54 to .93. Between group comparisons indicated that few differences were statistically significant out of the 48 comparisons (see Table 1). The authors suggested that the indicated differences might be chance observations. We also suggest that some statistically significant differences were not clinically meaningful because they were between correlations that both indicated temporal reliability.

Limitations of the study include that the college sample might not generalize to other population segments (e.g., adults, children, etc.). Also, a one week interval might not be long enough to reduce possible memory effects. Another limitation is that the researchers dismissed some significant test-retest differences that were quite large; for example, the AUDIT dependence scale. Finally, the study utilized a self-report methodology, which could mean participants consistently presented the same inaccurate information on both tests, yielding acceptable reliability estimates at the cost of validity.

This study shows the promise of online survey techniques, and researchers might glean various benefits by using this technology. Important benefits might include: cutting research costs, reaching more and different people, and reducing time and travel restrictions on potential research participants. Researchers should continue to study this methodology to further understand its benefits and costs.

What do you think? Click Here and let us know! Comments can be addressed to Leslie Bosworth.

References       

Babor, T. F., De la Fluente, J. R., Saunders, J., & Grant, M. (1992). The alcohol use disorders identification test: Guidelines for use in primary health care. Geneva, Switzerland: World Health Organization.

McConnaughy, E. A., DiClemente, C. C., Prochaska, J. O., & Velicer, W. F. (1993). Stages of change in psychotherapy: Measurement and sample profiles. Psychotherapy: Theory, Research and Practice, 20, 368-375.

Miller, E. T. (1997). Predicting successful self-initiated health-related behavior change in the context of New Year's resolutions: Utilizing the Internet for survey research. University of Washington.

Miller, E. T., Neal, D. J., Roberts, L. J., Baer, J. S., Cressler, S. O., Metrik, J., & Marlatt, G. A. (2002). Test-retest reliability of alcohol measures: Is there a difference between Internet-based assessment and traditional methods? Psychology of Addictive Behaviors, 16(1), 56-63.

Schmidt, W. C. (1997). World Wide Web survey research: Benefits, potential problems, and solutions Behavior Research Methods, Instruments and Computers, 29, 274-279.

Skinner, H. A., & Allen, B. A. (1982). Alcohol dependence syndrome, measurement and validation. Journal of Abnormal Psychology, 91, 199-209.

White, H. R., & Labouvie, E. W. (1989). Towards the assessment of adolescent problem drinking. Journal of Studies on Alcohol, 50, 30-37.

Comments

[Catherine Mbinya]

Quite informative.

yes correct don`t be a technology addict. it spoil our brain also increase the chances to get depressed harmful affect on our body posture and eyes.

[couponstechie]

i am quitely satisfied with this it ruins our brain we have to take some actions regarding this.

[eunah]

unfortunately most of the millennials are technology addicts and if something is not done it will only get worse

[Simon Kiplangat Bett]

With technology, everything is becoming easy and fast

[Simon Kiplangat Bett]

The trend is there for everyone to see. we are now dependent on technology and it is going to get worse

[chrispinus Kulenya]

I like the fact that reducing the extent that participants bias their self-presentation can benefit addiction-related research. Very informative
Keep up

[Stephen Kiboi]

This is an eye opener.

[jenny]

Great article. Couldn’t be written much better!