Disentangling Administration Errors From Scale Development Errors in Survey Research
DOI:
https://doi.org/10.33423/jmdc.v18i1.6820Keywords:
marketing, development, survey design, information entropy, scale development, Tobit regressionAbstract
In a recent manuscript, Friesner, et al. (2023) used the concept of information entropy to assess the quantity of information in survey responses. They demonstrate how assessments of the quantity of information can be used to identify possible errors in a survey’s administration. A major limitation of their methodology is that it assumes the survey items used to elicit consumer preferences were created appropriately and contained a meaningful quantity of information. The current study addresses this limitation by incorporating a methodology developed by Friesner et al. (2021) into the Friesner et al. (2023) methodology. The combined methodology is applied to the same data studied in both Friesner et al. (2021) and Friesner et al. (2023), which allows for a direct comparison of the quantity of information gained/lost from survey administration versus scale development. The results indicate that the survey used in the empirical application exhibits flaws in both scale design and survey administration.
References
Abd Gani, N., Rathakrishnan, M., & Krishnasamy, H. (2020). A pilot test for establishing validity and reliability of qualitative interview in the blended learning English proficiency course. Journal of Critical Reviews, 7(5), 140–143. DOI: https://doi.org/10.31838/jcr.07.05.23
Ballard, A. (2019). Framing bias in the interpretation of quality improvement data: Evidence from an experiment. International Journal of Health Policy and Management, 8(5), 307–314. DOI: https://doi.org/10.15171/ijhpm.2019.08
Batra, R., Homer, P., & Kahle, L. (2001). Values, susceptibility to normative influence, and attribute importance weights: A nomological analysis. Journal of Consumer Psychology, 11(2), 115–128. DOI: https://doi.org/10.1207/153276601750408348
Beatty, S., Kahle, L., Homer, P., & Misra, S. (1985). Alternative management approaches to consumer values: The list of values and the Rokeach value survey. Psychology and Marketing, 2(3), 181–200. DOI: https://doi.org/10.1002/mar.4220020305
Bozman, C.S., Kurpis, L.V., & Frye, C. (2010). Hoopfest: Using longitudinal economic impact data to assess the success of a strategic reorientation. Sport Management Review, 13, 65–81. DOI: https://doi.org/10.1016/j.smr.2009.04.007
Bradley, K., Peabody, M., & Sampson, S. (2015). Quality control in survey design: Evaluating a survey of educators’ attitudes concerning differentiated compensation. International Journal of Assessment Tools in Education, 2(1), 3–21. DOI: https://doi.org/10.21449/ijate.239565
Carré, A., Stefaniak, N., D’Ambrosio, F., Bensalah, L., & Besche-Richard, C. (2013). The Basic Empathy Scale in Adults (BES-A): Factor structure of a revised form. Psychological Assessment, 25(3), 679–691. DOI: https://doi.org/10.1037/a0032297
Dahl, F., & Osteras, N. (2010). Quantifying information content in survey data by entropy. Entropy, 12(2), 161–163. DOI: https://doi.org/10.3390/e12020161
Deming, W.E. (1944). On errors in surveys. American Sociological Review, 9(4), 359–369. DOI: https://doi.org/10.2307/2085979
Dillman, D.A. (2000). Mail and internet surveys: The tailored design method. New York, NY: John Wiley and Sons.
Draugalis, J., Coons, S., & Plaza C. (2008). Best practices for survey research reports: A synopsis for authors and reviewers. American Journal of Pharmaceutical Education, 72(11), Article 11. Retrieved from https://www.ajpe.org/content/72/1/11 DOI: https://doi.org/10.5688/aj720111
Entman, R. (2007). Framing bias: Media in the distribution of power. Journal of Communication, 57(1), 163–173. DOI: https://doi.org/10.1111/j.1460-2466.2006.00336.x
Fenner, K., Hyde, M., Crean, A., & McGreevy, P. (2020). Identifying sources of potential bias when using online survey data to explore horse training, management, and behaviour: A systematic literature review. Veterinary Sciences, 7(3), Article 140. https://doi.org/10.3390/vetsci7030140 DOI: https://doi.org/10.3390/vetsci7030140
Friesner, D., Bozman, C., McPherson, M., Valente, F., & Zhang, A. (2021). Information entropy and scale development. Journal of Survey Statistics and Methodology, 9(5), 1183–1203. DOI: https://doi.org/10.1093/jssam/smaa034
Friesner, D., Bozman, C., McPherson, M., Valente, F., & Zhang, A. (2023). Information entropy as a quality control tool in survey research. Journal of Marketing Development and Competitiveness, 17(1), 46–61.
Friesner, D., Valente, F., & Bozman, C.S. (2016). Using entropy-based information theory to evaluate survey research. Journal of Marketing Development and Competitiveness, 10(3), 32–48.
Golan, A. (2006). Information and entropy econometrics – A review and Synthesis. Foundations and Trends in Econometrics, 2(1–2), 1–145. DOI: https://doi.org/10.1561/0800000004
Golan, A., Judge, G., & Miller, D. (1996). Maximum entropy econometrics: Robust estimation with limited data. New York, NY: John Wiley and Sons.
Golan, A., Judge, G., & Perloff, J. (1996). A maximum entropy approach to recovering information from multinomial response data. Journal of the American Statistical Association, 91(434), 841–853. DOI: https://doi.org/10.1080/01621459.1996.10476952
González-Cabrera, M., Ortega-Martínez, A., Martínez-Galiano, J., Hernández-Martínez, A., Parra-Anguita, L., & Frías-Osuna, A. (2020). Design and validation of a questionnaire on communicating bad news in nursing: A pilot study. International Journal of Environmental Research and Public Health, 17(2), Article 457. https://doi.org/10.3390/ijerph17020457 DOI: https://doi.org/10.3390/ijerph17020457
Hair, J., Black, W., Babin, B., Anderson, R., & Tatham, R. (2006). Multivariate data analysis (6th Ed.). Upper Saddle River, NJ: Pearson/Prentice Hall.
Hendra, R., & Hill, A. (2019). Rethinking response rates: New evidence of little relationship between survey response rates and nonresponse bias. Evaluation Review, 43(5), 307–330. DOI: https://doi.org/10.1177/0193841X18807719
Imbens, G., & Lancaster, T. (1996). Efficient estimation and stratified sampling. Journal of Econometrics, 74(2), 289–318. DOI: https://doi.org/10.1016/0304-4076(95)01756-9
Jaynes, E. (1957). Information theory and statistical mechanics. Physics Review, 106(4), 620–630. DOI: https://doi.org/10.1103/PhysRev.106.620
Jaynes, E. (1982). On the rationale of maximum-entropy methods. Proceedings of the IEEE, 70(9), 939–952. DOI: https://doi.org/10.1109/PROC.1982.12425
Johnson, T., Cho, Y., Holbrook, A., O’Rourke, D., Warnecke, R., & Chavez, N. (2006). Cultural variability in the effects of question design features on respondent comprehension of health surveys. Annals of Epidemiology, 16(9), 661–668. DOI: https://doi.org/10.1016/j.annepidem.2005.11.011
Jolliffe, D., & Farrington, D.P. (2006). Development and validation of the basic empathy scale. Journal of Adolescence, 29, 589–611. DOI: https://doi.org/10.1016/j.adolescence.2005.08.010
Jordan, P., & Troth, A. (2020). Common method bias in applied settings: The dilemma of researching in organizations. Australian Journal of Management, 45(1), 3–14. DOI: https://doi.org/10.1177/0312896219871976
Kahle, L., Beatty, S., & Homer, P. (1986). Alternative measurement approaches to consumer values: The list of values (LOV) and values and life style (VALS). Journal of Consumer Research, 13(3), 405–409. DOI: https://doi.org/10.1086/209079
Kahle, L.R. (1983). Social Values and Social Change: Adaptation to Life in America. New York, NY: Praeger.
Kurpis, L.V., Bozman, C.S., & Kahle, L.R. (2010). Distinguishing between amateur sport participants and spectators: The list of values approach. International Journal of Sport Management and Marketing, 7(3/4), 190–201. DOI: https://doi.org/10.1504/IJSMM.2010.032550
Marquis, K., Marquis, M., & Polich, J. (1986). Response bias and reliability in sensitive topic surveys. Journal of the American Statistical Association, 81(394), 381–389. DOI: https://doi.org/10.1080/01621459.1986.10478282
Schnell, L. (2014, June 27). Hoopfest, the world’s largest 3-3 tourney, turns 25 this weekend. Retrieved from https://www.si.com/college-basketball/2014/06/27/spokane-hoopfest-3-3-basketball-tournament-25-years
Shannon, C. (1948). A mathematical theory of communication. Bell System Technical Journal, 27, 37–423. DOI: https://doi.org/10.1002/j.1538-7305.1948.tb01338.x
Suchman, E.A. (1962). An analysis of “bias” in survey research. The Public Opinion Quarterly, 26(1), 102–111. DOI: https://doi.org/10.1086/267075
Tripathi, G. (2011). Generalized method of moments (GMM) based inference with stratified samples when the aggregate shares are known. Journal of Econometrics, 165(2), 258–265. DOI: https://doi.org/10.1016/j.jeconom.2011.08.004
van Teijlingen, E., & Hundley, V. (2001). The importance of pilot studies. Social Research Update, (35), 1–4. Retrieved from http://sru.soc.surrey.ac.uk/SRU35.pdf
van Teijlingen, E., Rennie, A.-M., Hundley, V., & Graham, W. (2001). The importance of conducting and reporting pilot studies: The example of the Scottish Births Survey. Journal of Advanced Nursing, 34(3), 289–295. DOI: https://doi.org/10.1046/j.1365-2648.2001.01757.x
Wikman, A., & Wärneryd, B. (1990). Measurement errors in survey questions: Explaining response variability. Social Indicators Research, 22(2), 199–212. DOI: https://doi.org/10.1007/BF00354840
Wing, C., Simon, K., & Bello-Gomez, R. (2018). Designing difference in difference studies: Best practices for public health policy research. Annual Review of Public Health, 39(1), 453–469. DOI: https://doi.org/10.1146/annurev-publhealth-040617-013507
Wiseman, F. (1972). Methodological bias in public opinion surveys. The Public Opinion Quarterly, 36(1), 105–108. DOI: https://doi.org/10.1086/267981
