Full Content is available to subscribers

Subscribe/Learn More  >

Crowd-Sourcing the Evaluation of Creativity in Conceptual Design: A Pilot Study

[+] Author Affiliations
Matthew Green

LeTourneau University, Longview, TX

Carolyn Conner Seepersad

University of Texas at Austin, Austin, TX

Katja Hölttä-Otto

Singapore University of Technology and Design, Singapore, Singapore

Paper No. DETC2014-34434, pp. V007T07A016; 12 pages
  • ASME 2014 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
  • Volume 7: 2nd Biennial International Conference on Dynamics for Design; 26th International Conference on Design Theory and Methodology
  • Buffalo, New York, USA, August 17–20, 2014
  • Conference Sponsors: Design Engineering Division, Computers and Information in Engineering Division
  • ISBN: 978-0-7918-4640-7
  • Copyright © 2014 by ASME


Creativity is often considered to be a critical aspect of engineering innovation and successful product design. Many methods have been proposed for enhancing creativity, originality, and innovation. When these methods are tested, the experiment often generates large numbers of concepts that must be evaluated by experts in a time-consuming process. Similarly, the increased use of crowd-sourcing for generating concepts often leads to a plethora of alternatives that must be evaluated. Accordingly, engineering design practitioners and researchers alike often find themselves evaluating large numbers of concepts. In this paper, the feasibility of using non-experts to evaluate engineering creativity is investigated. Dozens of students at two universities are asked to rate the originality of several different solutions to a design problem, for which validated expert ratings are available. Results indicate that it is possible to extract expert-level ratings from the non-expert student raters by focusing on the student raters with excellent inter-rater agreement amongst themselves and by training the students with example problems prior to the rating exercise. These results suggest that it may be possible to evaluate originality reliably with a large set of novice raters, perhaps with a Mechanical Turk type of approach.

Copyright © 2014 by ASME



Interactive Graphics


Country-Specific Mortality and Growth Failure in Infancy and Yound Children and Association With Material Stature

Use interactive graphics and maps to view and sort country-specific infant and early dhildhood mortality and growth failure data and their association with maternal

Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In