Tags

Type your tag names separated by a space and hit enter

Beyond subjective judgments: Predicting evaluations of creative writing from computational linguistic features.
Behav Res Methods. 2019 04; 51(2):879-894.BR

Abstract

The question of how to evaluate creativity in the context of creative writing has been a subject of ongoing discussion. A key question is whether something as elusive as creativity can be evaluated in a systematic way that goes beyond subjective judgments. To answer this question, we tested whether human evaluations of the creativity of short stories can be predicted by: (1) established measures of creativity and (2) computerized linguistic analyses of the stories. We conducted two studies, in which college students (with and without interest and experience in creative writing) wrote short stories based on a writing prompt. Independent raters (six in Study 1, five in Study 2) assessed the stories using an evaluation rubric specifically designed to assess aspects of creativity, on which they showed high interrater reliability. We provide evidence of convergent validity, in that the rubric evaluations correlated with established creativity measures, including measures of divergent thinking, associative fluency, and self-reported creative behavior and achievements. Linguistic properties of the short stories were analyzed with two computerized text analysis tools: Coh-Metrix, which analyzes aspects of text cohesion and readability, and Linguistic Inquiry and Word Count, which identifies meaningful psychological categories of the text content. Linguistic features predicted the human ratings of creativity to a significant degree. These results provide novel evidence that creative writing can be evaluated reliably and in a systematic way that captures objective features of the text. The results further establish our evaluation rubric as a useful tool to assess creative writing.

Authors+Show Affiliations

University of California, Santa Barbara, CA, USA. claire.zedelius@gmail.com.University of British Columbia, Vancouver, British Columbia, Canada.University of California, Santa Barbara, CA, USA.

Pub Type(s)

Journal Article
Research Support, Non-U.S. Gov't

Language

eng

PubMed ID

30264366

Citation

Zedelius, Claire M., et al. "Beyond Subjective Judgments: Predicting Evaluations of Creative Writing From Computational Linguistic Features." Behavior Research Methods, vol. 51, no. 2, 2019, pp. 879-894.
Zedelius CM, Mills C, Schooler JW. Beyond subjective judgments: Predicting evaluations of creative writing from computational linguistic features. Behav Res Methods. 2019;51(2):879-894.
Zedelius, C. M., Mills, C., & Schooler, J. W. (2019). Beyond subjective judgments: Predicting evaluations of creative writing from computational linguistic features. Behavior Research Methods, 51(2), 879-894. https://doi.org/10.3758/s13428-018-1137-1
Zedelius CM, Mills C, Schooler JW. Beyond Subjective Judgments: Predicting Evaluations of Creative Writing From Computational Linguistic Features. Behav Res Methods. 2019;51(2):879-894. PubMed PMID: 30264366.
* Article titles in AMA citation format should be in sentence-case
TY - JOUR T1 - Beyond subjective judgments: Predicting evaluations of creative writing from computational linguistic features. AU - Zedelius,Claire M, AU - Mills,Caitlin, AU - Schooler,Jonathan W, PY - 2018/9/29/pubmed PY - 2019/7/13/medline PY - 2018/9/29/entrez KW - Coh-Metrix KW - Creative writing KW - Creativity KW - Language KW - Linguistic inquiry and word count (LIWC) SP - 879 EP - 894 JF - Behavior research methods JO - Behav Res Methods VL - 51 IS - 2 N2 - The question of how to evaluate creativity in the context of creative writing has been a subject of ongoing discussion. A key question is whether something as elusive as creativity can be evaluated in a systematic way that goes beyond subjective judgments. To answer this question, we tested whether human evaluations of the creativity of short stories can be predicted by: (1) established measures of creativity and (2) computerized linguistic analyses of the stories. We conducted two studies, in which college students (with and without interest and experience in creative writing) wrote short stories based on a writing prompt. Independent raters (six in Study 1, five in Study 2) assessed the stories using an evaluation rubric specifically designed to assess aspects of creativity, on which they showed high interrater reliability. We provide evidence of convergent validity, in that the rubric evaluations correlated with established creativity measures, including measures of divergent thinking, associative fluency, and self-reported creative behavior and achievements. Linguistic properties of the short stories were analyzed with two computerized text analysis tools: Coh-Metrix, which analyzes aspects of text cohesion and readability, and Linguistic Inquiry and Word Count, which identifies meaningful psychological categories of the text content. Linguistic features predicted the human ratings of creativity to a significant degree. These results provide novel evidence that creative writing can be evaluated reliably and in a systematic way that captures objective features of the text. The results further establish our evaluation rubric as a useful tool to assess creative writing. SN - 1554-3528 UR - https://www.unboundmedicine.com/medline/citation/30264366/Beyond_subjective_judgments:_Predicting_evaluations_of_creative_writing_from_computational_linguistic_features_ L2 - https://dx.doi.org/10.3758/s13428-018-1137-1 DB - PRIME DP - Unbound Medicine ER -