What are the differences between formative evaluations and summative  evaluations? At which stage in product development do you believe that  evaluations should use controlled settings instead of natural settings?  Why?

Be sure to respond to at least one of your classmates’ posts.

student post

 Christopher Owens           

Greetings Dr. D and Class,

Formative evaluations are conducted during design or redesign  phases while summative evaluations are conducted to assess the success  of a finished product (1).   For example, when developing the first  iPad, formative evaluations were conducted to ascertain the usability  specifically focusing on apps over browser.  Summative evaluations were  completed a year later to help identify usability issues with apps and  websites and have more interaction between developers and the Apple iPad  development team as there was little previous interaction due to the  required secrecy about the products launch.

In regards to the stage controlled settings should be used versus  natural settings, I believe the answer depends on what question the test  is attempting to answer.  For example, field studies or in-the-wild  studies can help identify opportunities for new technology, help  establish requirements for a new design, or facilitate the introduction  of technology (1).  First, this is because controlled settings cannot  account for how people will react or use a product in their daily  lives.  Second, behaviors can change as people interact with a product  such as problem-solving strategies as they become more familiar and  comfortable with a tool (1).  Controlled settings are good at  discovering usability problems (1).  Returning to my opening comment  relating to what question the test is attempting to answer, there are  issues that can also influence the method of evaluation:

• Controlled settings have a high reliability of producing the  same results on different occasions while natural settings may not. 
• Ecological validity is concerned with how an environment can  influence the results such as the Hawthorne effect of when participants  are aware that they are being studied.  Lab experiments have a low  ecological validity while ethnographic studies have a high ecological  validity. 
• Bias can interfere with the results which can effect controlled and natural settings. 
• Time can influence the evaluation method.  
• Funding can influence the evaluation method. 

Overall, using a combination of methods across categories can help  designers better understand issues and opportunities such as usability  tests combined with observation in natural settings

Ulrich and Yang describe the following product development  process: planning, concept development, system level design, detail  design, testing and refinement, and production ramp-up (2).  I believe  natural settings can be useful in answering questions in planning  (trying to determine opportunities), concept development (developing  opportunities),  and production (observing how users naturally use a  product).  While controlled settings my be better at evaluating products  in the conceptualization, system level design, detail design, and  testing and refinement phases due to the specificity of goals as they  relate to features and functionality more centered around the usability  of the product being developed in its design stages (2).  All in all,  there are stages that can benefit from using a combination of methods to  help designers get a fuller understanding of the usability issues and  opportunities.  However, what method or methods that are used will be  determined by the questions that need to be answered (1).

1. Helen Sharp, Jennifer Preece, Yvonne Rogers. 2019. Interaction  Design, 5th Edition. Strayer University Bookshelf, (8th Edition). Wiley  Professional Development (P&T).

2. Karl T. Ulrich, Maria C. Yang. 2020. Product Design and Development, Seventh Edition.  McGraw-Hill.