Sensory Evaluation and Product
Concept Testing
Jean-François MeullenetProfessor of Sensory Science
and HeadDepartment of Food Science
Why are they important?
Most new product introductions fail (AC Nielsen)
Why do they fail? Consumers don’t / can’t buy them They don’t generate sufficient revenues
Why? not relevant / valuable not new / distinctive not fashionable / fit with consumer style
Factors contributing to Failure
Several factors relate to competencies and responsibilities lack of consumer value poor cooperation between functional groups (Marketing & R&D) too much inward (rather then outward) looking lack of consideration of real world competition up-front home work voice of the consumer (VOC) sharp, stable and early product definitions go / no go decision points / gates
Too conservative; too close to what is already out there methodology too much optimization – focused? lack of precise tools for outside the box?
Sensory and the product Development Process
Sensory and
consumer methods
Idea generation
Idea Screening
Concept creation
Business/ legal analysis
Product/ Marketing
development
Market Test
Commercialization
Reposition/ Extension/
Retire
Idea
Business feasibility
Performanceevaluation
Marketadoption
Sensory and the product Development Process
Sensory and
consumer methods
Idea generation
Idea Screening
Concept creation
Business/ legal
analysisProduct/ Marketing developm
ent
Market Test
Commercialization
Reposition/
Extension/ Retire
Focus Groups• Group discussions• 10-12 participants• Moderator• Guided Script• 1-2 hours
Consumer Interviews/ Surveys
• Greater number of participants
• Assess satisfaction with current products
• Identify consumer needs/ trends
From Delgado, 2010
Sensory in product development
Sensory and
consumer methods
Idea generation
Idea Screening
Concept creation
Business/ legal
analysisProduct/
Marketing development
Market Test
Commercialization
Reposition/
Extension/ Retire
Difference Tests
Can you discriminate between two very similar and confusable stimuli?
ObviouslyDifferent
Confusable
Discriminant Tests (e.g. triangle, duo-trio)
• Determine difference between two samples• e.g. Reformulation new vs. current ingredient• Product matching
Prototypes evaluation,comparison and optimization
From Delgado, 2010
Sensory in product development
Sensory and
consumer methods
Idea generation
Idea Screening
Concept creation
Business/ legal
analysisProduct/
Marketing development
Market Test
Commercialization
Reposition/
Extension/ Retire
Descriptive Analysis (e.g. Spectrum, Tragon, Generic)
• Sensory characterization of samples• Development of language to describe
products (lexicon)• Identification of differences/similarities among
products• Trained panelists
Prototypes evaluation,comparison and optimization
From Delgado, 2010
Sensory in product development
Sensory and
consumer methods
Idea generation
Idea Screening
Concept creation
Business/ legal
analysisProduct/
Marketing development
Market Test
Commercialization
Reposition/
Extension/ Retire
Hedonic Tests (e.g. preference, acceptance, intend to purchase)
• Understand consumer acceptance of products• Measure attitudes• Identification of segments• Selection of best prototypes or directions for
optimization
Prototypes evaluation,comparison and optimization
Market TestingFrom Delgado, 2010
Quantitative hedonic methods
Quantitative methods are used to evaluate the response of consumers and should include larger panels (75-500)
The methods vary widely and can be used to evaluate: Preference Overall degree of liking Liking for specific sensory
attributes (texture, flavor, appearance)
Diagnostics for sensory attributes (JAR)
Sensory and
consumer methods
Idea generation
Idea Screening
Concept creation
Business/ legal
analysisProduct/
Marketing developme
nt
Market Test
Commercialization
Reposition/ Extension/
Retire
Hedonic Scales
The 9-point verbal hedonic scale
Scale developed by the US Quartermaster Food and Container Institute in 1949. Described inPERYAM, D.R. and PILGRIM, F.J. 1957. Hedonic scale method of measuring food preferences. Food Technol. 11(September), 9–14.
What is your overall impression of this product?
Overall consumer impression of products usually quantified on hedonic scale• Hedonic ratings only because
consumers cannot explain what they like
• Others have consumers rate specific attributes for intensity or appropriateness
Intensity or appropriateness ratings
used to explain consumer hedonic ratings
Diagnostic scaling
Diagnostic Scales
Just Right Scales
Implied Just Right
Scales
Choice of test
Monadic (marketing) or Sequential Monadic (R&D)
A B C DA
B
C
D B C D A
C D A B
D A B C
Sequential monadic allows the identification of consumer segments and the opportunity for product optimization
No presentation order effect
Methods for Optimization
How sensory attributes drive liking
Liking vs. sensory level Quadratic function Identify optimal sensory
level Target landmarks for
development
4
4.5
5
5.5
6
6.5
7
7.5
0 1 2 3
Liki
ng
Saltiness
Optimum
Current
Sensory segments: How do you identify a true opportunity?
Consumers show different liking patterns
Indentify preference segments
Discover new niches
4
4.5
5
5.5
6
6.5
7
7.5
0 1 2 3
Liki
ng
Saltiness
Optima
Seg 1
Seg 2
Total
Preference Mapping
Preference mapping category of methods understand the relationship
between product liking and their sensory properties
Can be used for several purposes including improving or optimizing existing products
Mapping methods yield a graphical representation of consumer preference and/or sensory
differences for a set of products
• Some competitor products• Some potential prototypes
Consumers evaluate 6 or more products
External versus Internal Preference Mapping
Concepts
18
Internal External
Set of competitive
products
Trained panelists describe the products
in sensory terms
Sensory Profiles
Hedonic Scores
Consumer panel assesses the products for
liking
Context of Preference Mapping
Statistical modelingK=number of products
T=number of sensory attributesN=number of consumers
S1 S2 … St-1 St
P1
P2
.
.
.
Pk
C1 C2 … Cn-1 Cn
P1
P2
.
.
.
Pk
From P. Schlich
Internal vs. ExternalInternal preference analysis• Stimulus location based on
liking (hedonic data drives orientation of the map)
• Sensory attributes can be fitted into preference space afterwards
• First dimension explains maximum variability in hedonic data
External preference analysis• Stimulus locations based on
similarity in sensory properties (sensory data drive orientation of the map)
• Preference data can be fitted into fixed space afterwards
• First Dimension explains maximum variance in sensory attribute descriptions
Mapping perceptions or preferences?
Perceptual mapPreference map
Basics on Internal Preference Mapping
IPM Data
Derives a multidimensional representation of products and consumers according to preference patterns
Data used is liking data (e.g. 9-pt hedonic scale scores for overall impression)
OLij
Consumers (j,k)
Pro
duct
s (i,
n)
IPM Data
ConsumersC1 C2 C3 C4 C5 C6 ……… C75 C76 C77 C78 C79 C80
Pro
duct
s
BYW 8 7 5 8 6 4 ……… 5 8 5 6 6 8GMG 3 7 6 6 4 8 ……… 5 4 4 5 4 9GUY 6 7 8 4 5 4 ……… 7 5 5 4 4 8MED 3 6 6 9 4 4 ……… 4 4 5 4 6 6MST 9 8 9 7 8 6 ……… 8 7 3 6 6 8MTR 4 7 9 4 7 8 ……… 8 7 4 7 4 9OCF 4 6 6 1 4 6 ……… 4 7 . 7 2 9SAN 7 7 5 8 8 6 ……… 5 7 8 7 7 7TBS 8 8 8 6 7 4 ……… 9 5 6 7 4 7TOM 7 7 8 6 5 7 ……… 3 9 5 6 7 7TRS 2 7 8 7 5 6 ……… 6 9 7 7 5 8
From White Corn Tortilla Chips pages 9-18 in Multivariate and Probabilistic Analyses of Sensory Science Problems, Meullenet et al. 2007
OLij
Consumers (j,k)
Pro
duct
s (i,
n)
MDPREF
-10
-8
-6
-4
-2
0
2
4
6
8
-15 -10 -5 0 5 10 15
PC2
(17.
48%
)
PC1 (27.75%)
Direction of increased liking
Each dot represents a consumer
Shows significant segmentation of consumers
P1
P2
P3
P4
P5
P6
Identifying Segments: Cluster analysis
A group of multivariate techniques whose primary purpose is to group objects based on the characteristics they possess.
Groups objects (respondents, products, variables, etc.) so that each object is similar to the other objects in the cluster and different from objects in all the other clusters.
Between-Cluster Variation = Maximize
Within-Cluster Variation = Minimize
Basic Concept
High
LowLow High
Freq
uenc
y of
eat
ing
out
Frequency of going to fast food restaurants
A Simple example
From Oupadissakoon and Dooley, 2008
High
Low
Low HighFrequency of going to fast food restaurants
Freq
uenc
y of
eat
ing
out
A Simple example
From Oupadissakoon and Dooley, 2008
High
LowLow HighFrequency of going to fast food restaurants
Freq
uenc
y of
eat
ing
out
A Simple example
From Oupadissakoon and Dooley, 2008
Effect of Centering
For the raw data, the pattern of preference is the same
Scaling effect not very interesting
For centered data P3 is favorite for 1 cluster and P1-P2 favorite for the other. More interesting
0
1
2
3
4
5
6
7
8
P1 P2 P3 P4 P5
Raw
Mean C1-2
Mean C3-4
0
1
2
3
4
5
6
7
8
P1 P2 P3 P4 P5
Centered
Means C1-3
Means C2-4
Internal Preference Mapping
Strengths Consumer data is the
starting point of the analysis..after all, it is called preference mapping
Allows the representation of consumer segmentation
Weaknesses The consumer data can
be poorly represented by first 2-3 dimensions
Consumers may like products equally but products are different sensorily (Optimization?)
Not as fully researched as external mapping
Basics on External Preference Mapping
S1 S2 … St-1 St
2.5 10.3 2.1 7.0 10.1
3.5 9.5 0.8 7.2 13.5
10.0 8.8 1.1 7.5 14.2
5.2 6.6 2.0 7.2 14.5
1.9 13.6 1.3 7.4 10.5
6.7 14.1 1.5 7.5 10.8
4.1 12.4 1.1 6.2 11.6
6.8 10.3 2.1 7.0 13.4
7.0 7.8 5.1 6.8 12.9
9.0 9.9 0.2 6.9 12.1
2.5 2.2 0.5 7.4 13.5
1.3 7.7 1.0 7.5 14.5
1.8 6.3 0.6 7.1 10.9
Sensory Space
Dim
ensi
on 2
Dimension 1
Principal Component
Analysis
strawberrysweet
thick
smooth
creamy
dairysour
Fitting consumers in the sensory space
Quadratic Surface Model
Vector Model
Circular Model
Elliptical Model
∑ ∑ ∑++=i i ij
jiijiiii PCPCcPCbPCaOL 2
∑=i
ii PCaOL
∑ ∑ =+=i i
itconsiii PCbPCaOL 2tan
∑ ∑+=i i
iiii PCbPCaOL 2
36
Four Types of Consumers
0PC1
PC2
F6
non-discrim
inator
0PC1
PC2
F7
Expr
essin
g D
islik
e
minimum0
PC1
PC2
F38
Expressing Liking
optimum
0PC1
PC2
F21
Ecle
ctic
optima
External Maps: Ideals?
-50.
00
-44.
00
-38.
00
-32.
00
-26.
00
-20.
00
-14.
00
-8.0
0
-2.0
0
4.00
10.0
0
16.0
0
22.0
0
28.0
0
34.0
0
40.0
0
46.0
0
-32.00
-26.00
-20.00
-14.00
-8.00
-2.00
4.00
10.00
16.00
22.00
28.00
34.00
40.00
46.00
52.00
58.00
64.00
Z
Axis 1 (27.75%)
Axi
s 2
(17.
48%
)
180-190170-180160-170150-160140-150130-140120-130110-120100-11090-10080-9070-8060-7050-60Braeburn
Fuji
Gibson's Green
Golden Delicious
Granny Smith
Johnson's Red
Pink Lady
Royal Gala
Sun Gold
Top Red
Known as the Danzartmethod
Surface response model used to fit individual consumers
Acceptable area for individuals identified based on predicted values
All consumer overlaid=density of satisfied consumers
External Preference Mapping
Strengths Very well researched Nice maps including
density maps of satisfied consumers (Danzart)
Weaknesses Representation of
products based on sensory properties not necessarily driving liking
Resulting in poor consumer fit
Use of quadratic models tend to overfit data
Alternative Data
Consumer intensity ratings, cata
Quantitative consumer
research is a key tool in
R&D
Overall consumer impression of products usually quantified on hedonic scale• Hedonic ratings
only because consumers cannot explain what they like
• Others have consumers rate specific attributes for intensity or appropriateness
Traditionally, sensory properties have been measured by trained panelists
Intensity or appropriateness ratings used to explain consumer hedonic ratings
INTRODUCTION
Set of competitive
products
Trained panelists describe the products
in sensory terms
Sensory Profiles
Hedonic Scores
Consumer panel assesses the products for
liking
The OP&P way!
Statistical modelingK=number of products
T=number of sensory attributesN=number of consumers
S1 S2 … St-1 St
P1
P2
.
.
.
Pk
C1 C2 … Cn-1 Cn
P1
P2
.
.
.
Pk
Consumers describe the products in sensory terms
Van Trijp, Punter, Mickartz and Kruithof, 2007, FQP 18:729-740
Consumer Intensity Ratings
Preference mapping with consumer sensory profiling
Sensory profiling with consumers actually works! An orange juice external preference shows that consumer liking scores were well fitted (R2=0.73) to a sensory space derived from consumer intensity ratings
AlwaysSaveBestChoice
FlNatural
GreatValue
MinMaid
OrganicValley
SimplyOrange
TropPremTropValencia Opt
Stability of Consumer data
This plot represents consumers’ stated ideal products based on ideal intensity ratings given to 9 orange juices by 100 consumers
Ideal intensity ratings vary by consumers but seem to be reproducible across the samples tested
This clearly shows that consumers have the ability to rate intensities in a reproducible manner
Ballots with intensity ratings can become very cumbersome when many products are evaluated
Intensity scaling can be a difficult concept for consumers
Intensity or appropriateness questions can have an impact on hedonic ratings
HOWEVER…
CATA is a compromise between only liking and asking intensity ratings
CATA
Previous CATA research
Type of methodology allowing a more instinctive description of the main sensory properties of the product tested
Type of question which can be about attributes, product usage or concept fit
Advantages and uses of check-all-that-apply response compared to traditional scaling of attributes for salty snacks J. Adams, A. Williams, B. Lancaster, M. Foley; Frito Lay, USA Rose Marie Pangborn Sensory Science Symposium, 2007
Product Attributes
Concept Deliverables
Occasion/Use
SweetSaltyCreamySoftTough
IndulgentEnergizingComfortingArtificialBland
As a mealAs a snackWhile drivingWatching TVAfter exercising
Multiple Factor Analysis
The counts for each of 13 ice cream
attributes in a check-all-that-apply question were
compared to the descriptive profiles via Multiple Factor Analysis (MFA),
using FactoMineR in R ©2008, v.2.6.2. -1.0 -0.5 0.0 0.5 1.0
-1.0
-0.5
0.0
0.5
1.0
Correlation circle
Dim 1 (31.96 %)
Dim
2 (1
9.24
%
)
DescriptiveConsumer
Sweet
Bitter
Sour
Salt
Vanilla.VanillinCooked.Milk
Milky
Butterfat
NFDM
Caramelized
Oxidized
Woody.Stick
Metallic
Astringent
Scoopability
Hardness.Oral
Denseness
Degree.of.Ice
Smoothness
Rate.of.MeltMouthcoat
Scoopability2
Elasticity
Gummy
HardSoft
Milk.dairy.flavor
Sweet
Buttery
Creamy.Smooth
Icy
Creamy.flavor
Natural.Vanilla
Artificial.VanillaCorn.Syrup
Custard.Eggy.
MFA: CATA and Descriptive
Overall good agreement
between sensory and CATA profiles
e.g. Soft hard opposite and
sensory hardness perpendicular
Agreement for icy, sweet, buttery, soft
and rate of melt-1.0 -0.5 0.0 0.5 1.0
-1.0
-0.5
0.0
0.5
1.0
Correlation circle
Dim 1 (31.96 %)
Dim
2 (1
9.24
%)
DescriptiveCATA
Sweet
Bitter
Sour
Salt
Vanilla.Vanillin
Cooked.Milk
Milky
Butterfat
NFDM
Caramelized
Oxidized
Woody.Stick
Metallic
Astringent
Scoopability
Hardness.Oral
Denseness
Degree.of.Ice
Smoothness
Rate.of.Melt
Mouthcoat
Scoopability2
Elasticity
Gummy
HardSoft
Milk.dairy.flavor
Sweet
Buttery
Creamy.Smooth
Icy
Creamy.flavor
Natural.Vanilla
Artificial.Vanilla Corn.Syrup
Custard.Eggy.
MFA on product & ideal configurations
Product configurations not identical but similar
Ideal location fairly stable
and closest to F and A
Greatest disagreement
among methods for products J, F
and A-4 -2 0 2
-4
-2
0
2
Individual factor map
Dim 1 (46.32 %)
Dim
2 (3
4.24
%)
A
BC
DE
F
G
HI
J
Ideal
DescriptiveCATAInternal
CATA based external map
Group Ideal closest to products Blue
Bell and Best Choice
78% of consumers satisfied by the
Ideal
Average R2 for consumers =0.61
Descriptive based external map
Ideal Point closest to
Eddy’s
Average R2=0.59 for consumers
76% of consumers
satisfied by this ideal
• Intensity rating given by consumers are reproducible
• CATA attribute data applied to preference mapping gave similar results to internal and external preference mapping
Consumers can be used to describe
food products and data used for
preference mapping studies
ALTERNATIVE METHODS
Alternative Modeling Methods
Ideal Point Modeling, Unfolding (Prefscal)
Ideal Point Mapping
R² = 0.9608
02468
1012
0 5 10
Dis
tanc
es to
Idea
l
Hedonic Scores
Individual consumer Ideal
Group Ideal
Correlation between the hedonic scores and the product distances can easily be calculated
Correlation should be negative: ideal closest to products with high OL
Significance of the correlation can be assessed (defined by α1=0.05)
Correlations for the neighboring points in the grid are statistically (defined by α2) compared to minimum correlation
If correlation not different from the minimum correlation this second location is added to the acceptable region of the sensory space for a consumer
Individual maps overlaid to yield final density mapPoint of maximum density in the sensory space taken as group ideal
Unfolding
The unfolding model is a geometric model to deal with preference and choice data
Locates individuals and alternatives (products) in a joint space
Has the advantage of representing individual ideals in a geometric fashion Useful for defining ideal product
nxp nxn
pxp pxn
Unfolding degenerate solution
BraeburnFuji
GibsonGolden Delicious
Granny Smith
Johnson's
Pink LadyRoyal Gala
Sun Gold Top Red
-8
-6
-4
-2
0
2
4
6
8
-8 -6 -4 -2 0 2 4 6 8 10
Prefscal
Busing, Groenen and Heiser (2005) proposed a penalty approach and succeeded in avoiding degeneracy
Braeburn
Fuji
Gibson
Golden Delicious
Granny Smith
Johnson's
Pink Lady
Royal Gala
Sun Gold
Top Red
-6
-4
-2
0
2
4
6
-8 -6 -4 -2 0 2 4 6
PCA vs. Prefscal
Based on product configurations from PCA and Prefscal
EDIPM applied Ideal point defined as that
minimizing the correlation between distances to the objects and hedonic scores (Rmin)
PREFSCAL allows to better fit consumers as Rmindistribution is shifted to greater |Rmin|
Better estimation of group ideal and ideal sensory profiles
-1-0.500.51
Rmin
PCA 2D EDIPM
PCA 3D EDIPM
UNFOLDING 3D EDIPM
Ideal Points: Unfolding is best!
-0.90
-0.80
-0.70
-0.60
-0.50
-0.40
-0.30
-0.20
-0.10
0.00
Ave
rage
Rm
in
Internal/PCA/3D
Internal/PCA/2D
External/PCA/3D
External/PCA/2D
Internal/Unfolding/3D
Unfolding/3D -0.7311 t-Ratio -7.65254
Internal/PCA/3D -0.7012 DF 1639
Mean Difference -0.0299 Prob > |t| <.0001
Std Error 0.0039
N 1650
Products and Ideals locationsAdding a third dimension to preference mapping problems creates problems with the representation of ideal points and group ideals
Consumer ideal points and products represented in a 3 dimensional scatter plot
Lacks actionability
Representation of 3D maps
• Adding a 3rd dimension complicates the representation of maps, especially when density of likers are overlaid on the maps
• The same representations can be achieved using either the internal or external framework
• This makes internal maps much more user friendly and equivalent in output quality to external maps
Ideal Vanilla?
Test Product
Test RegionBBell
BBunny
Ben&Jer
Best Choice
Bryers
EddysGreatValue
GuiltFree
Hdazs
Yarnell
Bitter
SourButterfat
Metallic
Degree of Ice
Smoothness
Mouthcoat
-0.6
-0.5
-0.4
-0.3
-0.2
-0.1
0.0
0.1
0.2
0.3
0.4
0.5
-0.6 -0.4 -0.2 0.0 0.2 0.4 0.6
Ice Cream LSA
Dim
2
Dim1
Attribute LSA Density Plot
Sweet 8.96 8.31
Bitter 0.07 0.02
Sour 0.02 0.00
Salt 0.66 0.66
Vanilla 5.70 5.16
Cooked Milk 2.40 2.08
Milky 3.06 3.20
Butterfat 4.46 4.56
NFDM 0.12 0.15
Caramelized 2.35 1.82
Oxidized 0.63 0.56
Woody 0.95 0.43
Metallic 0.25 0.20
Astringent 4.12 3.72
Scoopability 10.44 10.5
Hardness 8.68 8.62
Denseness 4.37 4.62
Degree of Ice 1.49 1.31
Smoothness 9.60 9.90
Rate of Melt 6.84 6.93
Mouth-coat 2.46 2.43
Elasticity 0.43 0.62
Sweet
Bitter Sour
Salt
Vanilla
Cooked Milk
Milky
Butterfat
NFDM
Caramelized
OxidizedWoodyMetallic
Astringent
Scoopability
Hardness
Denseness
Degree of Ice
Smoothness
Rate of Melt
Mouth-coat
Elasticity
R² = 0.9937
-2
0
2
4
6
8
10
12
0 2 4 6 8 10 12
Den
sity
Plo
t Ide
al P
oint
Pro
file
LSA Ideal Point Profile
Comparison of Best Choice to Optimal Profile
Attribute Best Choice Optimal
Sweet 8.50 8.31
Bitter 0.07 0.02
Sour 0.00 0.00
Salt 0.94 0.66
Vanilla/Vanillin 5.26 5.16
Cooked Milk 2.08 2.08
Milky 3.23 3.20
Butterfat 4.38 4.56
NFDM 0.27 0.15
Caramelized 2.32 1.82
Oxidized 1.04 0.56
Woody Stick 0.28 0.43
Metallic 0.20 0.20
Astringent 4.05 3.72
Scoopability 10.43 10.50
Hardness-Oral 8.61 8.62
Denseness 4.18 4.62
Degree of Ice 2.23 1.31
Smoothness 8.13 9.90
Rate of Melt 6.38 6.93
Mouthcoat 2.30 2.43
Elasticity 0.24 0.62
Final Thoughts
It is about the consumer! They are hard to understand You don’t often have time to understand them We need to make time We need to become more sophisticated in the use
of sensory and consumer tool
Even when we develop vanilla ice cream
Muchas Gracias!
Recommended