Full Content is available to subscribers

Subscribe/Learn More  >

Study of Existing Metrics Used in Measurement of Ideation Effectiveness

[+] Author Affiliations
Ramesh Srivathsavai, Nicole Genco, Katja Hölttä-Otto

University of Massachusetts Dartmouth, North Dartmouth, MA

Carolyn C. Seepersad

The University of Texas at Austin, Austin, TX

Paper No. DETC2010-28802, pp. 355-366; 12 pages
  • ASME 2010 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
  • Volume 5: 22nd International Conference on Design Theory and Methodology; Special Conference on Mechanical Vibration and Noise
  • Montreal, Quebec, Canada, August 15–18, 2010
  • Conference Sponsors: Design Engineering Division and Computers in Engineering Division
  • ISBN: 978-0-7918-4413-7 | eISBN: 978-0-7918-3881-5
  • Copyright © 2010 by ASME


In recent years, many new idea generation methods have been developed to generate innovative concepts. The effectiveness of those methods is evaluated by applying a set of metrics to the resulting concepts. Several metrics have been proposed for this purpose, including quality, novelty, and variety metrics, but the inter-rater reliability of those metrics has not been investigated extensively. In this paper, the inter-rater reliability of three existing metrics is analyzed by applying them to the results of a representative idea generation study. The effects on inter-rater agreement of analyzing concepts at the overall concept level versus the feature level are investigated, along with the impacts of alternative scales for specific metrics. In general, the inter-rater reliability of the metrics is found to be relatively low, with the most reliable results obtained at the feature level. The use of different scales also affects inter-rater reliability, but the effect is less significant. In addition to their low levels of repeatability, the metrics differ in how novelty is appraised.

Copyright © 2010 by ASME
Topics: Reliability



Interactive Graphics


Country-Specific Mortality and Growth Failure in Infancy and Yound Children and Association With Material Stature

Use interactive graphics and maps to view and sort country-specific infant and early dhildhood mortality and growth failure data and their association with maternal

Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In