Week 10: DOE & Power Analysis

EMSE 6035: Marketing Analytics for Design Decisions

John Paul Helveston

October 30, 2024

1 / 33

Quiz 4

10:00

Link is in the #class channel

2 / 33

Week 10: DOE & Power Analysis1. Design of Experiment2. Design Efficiency3. Power Analysis3 / 33

Week 10: DOE & Power Analysis1. Design of Experiment2. Design Efficiency3. Power Analysis4 / 33

Main & Interaction Effects5 / 33

Full design space for 3 effects: A, B, C

6 / 33

Full design space for 3 effects: A, B, C

Example: Cars

A: Electric? (Yes+ or No-)

B: Warranty? (Yes+ or No-)

C: Ford? (Yes+ or No-)

7 / 33

Main Effects

$ME(a) =$

$\left( \frac{A + AB + AC + ABC}{4}\right) -$

$\left( \frac{I + B + C + BC}{4}\right)$

(A: Electric? Yes+ or No-)

8 / 33

Interaction Effects

$INT(ab) =$

$\frac{1}{2}\left[ \left( \frac{AB + ABC}{2}\right) - \left( \frac{B + BC}{2}\right) \right] -$

$\frac{1}{2}\left[ \left( \frac{A + AC}{2}\right) - \left( \frac{I + C}{2}\right) \right]$

9 / 33

Example: Wine Pairings

meat
wine


fish
white 

fish
red

steak
white 

steak
red


10 / 33

meat	wine
fish	white
fish	red
steak	white
steak	red

Example: Wine Pairings

meat
wine


fish
white 

fish
red

steak
white 

steak
red


Main Effects
meat: Fish or Steak?
wine: Red or White?

10 / 33

meat	wine
fish	white
fish	red
steak	white
steak	red

Example: Wine Pairings

meat
wine


fish
white

fish
red

steak
white

steak
red


Main Effects
meat: Fish or Steak?
wine: Red or White?

Interaction Effects
meat*wine: Red or White wine with Steak?
meat*wine: Red or White wine with Fish?

11 / 33

meat	wine
fish	white
fish	red
steak	white
steak	red

Open interactions.qmd12 / 33

Fractional vs Full Factorial Designs13 / 33

Full Factorial DesignExample: Cars
A: Electric? (Yes+ or No-)
B: Warranty? (Yes+ or No-)
C: Ford? (Yes+ or No-)
library(cbcTools)
profiles <- cbc_profiles(
    electric = c(1, 0),
    warranty = c(1, 0),
    ford     = c(1, 0)
)
profiles

#>   profileID electric warranty ford
#> 1         1        1        1    1
#> 2         2        0        1    1
#> 3         3        1        0    1
#> 4         4        0        0    1
#> 5         5        1        1    0
#> 6         6        0        1    0
#> 7         7        1        0    0
#> 8         8        0        0    0
14 / 33

Full Factorial Design

Balanced?

All levels appear an equal number of times.

Orthogonal?

All pairs of levels appear together an equal number of times.

library(cbcTools)
profiles <- cbc_profiles(
    electric = c(1, 0),
    warranty = c(1, 0),
    ford     = c(1, 0)
)
profiles

#>   profileID electric warranty ford
#> 1         1        1        1    1
#> 2         2        0        1    1
#> 3         3        1        0    1
#> 4         4        0        0    1
#> 5         5        1        1    0
#> 6         6        0        1    0
#> 7         7        1        0    0
#> 8         8        0        0    0

15 / 33

Fractional Factorial Design

Balanced?

All levels appear an equal number of times.

Orthogonal?

All pairs of levels appear together an equal number of times.

profiles[c(1, 3, 5, 6),]

#>   profileID electric warranty ford
#> 1         1        1        1    1
#> 3         3        1        0    1
#> 5         5        1        1    0
#> 6         6        0        1    0

16 / 33

Comparing Full and Fractional Factorial DesignsOpen balance-orthogonality.qmd17 / 33

Practice Question 1

Consider the following experiment design

a	b	c	Effect
+	-	-	A
-	+	-	B
+	-	+	AC
-	+	+	BC

a) Is the design balanced? Is is orthogonal?

b) Write out the equation to compute the main effect for a, b, and c.

c) Are any main effects confounded? If so, what are they confounded with?

18 / 33

Week 10: DOE & Power Analysis1. Design of Experiment2. Design Efficiency3. Power Analysis19 / 33

A simple conjoint experiment about cars

Attribute	Levels
Brand	GM, BMW, Ferrari
Price	$20k, $40k, $100k

Design: 9 choice sets, 3 alternatives each

20 / 33

A simple conjoint experiment about cars

Attribute	Levels
Brand	GM, BMW, Ferrari
Price	$20k, $40k, $100k

Design: 9 choice sets, 3 alternatives each

Attribute counts:
brand:
  GM   BMW  Ferrari
  10    11    6
price:
 20k  40k 100k
  9    9   9

20 / 33

A simple conjoint experiment about cars

Attribute	Levels
Brand	GM, BMW, Ferrari
Price	$20k, $40k, $100k

Design: 9 choice sets, 3 alternatives each

Attribute counts:
brand:
  GM   BMW  Ferrari
  10    11    6
price:
 20k  40k 100k
  9    9   9

Pairwise attribute counts:
brand & price:
          20k 40k 100k
  GM        3   0    7
  BMW       4   5    2
  Ferrari   2   4    0

20 / 33

A simple conjoint experiment about cars

Attribute	Levels
Brand	GM, BMW, Ferrari
Price	$20k, $40k, $100k

Design: 90 choice sets, 3 alternatives each

21 / 33

A simple conjoint experiment about cars

Attribute	Levels
Brand	GM, BMW, Ferrari
Price	$20k, $40k, $100k

Design: 90 choice sets, 3 alternatives each

Attribute counts:
brand:
  GM    BMW   Ferrari
  92    80     98
price:
  20k  40k 100k
  91   84   95

21 / 33

A simple conjoint experiment about cars

Attribute	Levels
Brand	GM, BMW, Ferrari
Price	$20k, $40k, $100k

Design: 90 choice sets, 3 alternatives each

Attribute counts:
brand:
  GM    BMW   Ferrari
  92    80     98
price:
  20k  40k 100k
  91   84   95

Pairwise attribute counts:
brand & price:
          20k 40k 100k
  GM      31  31  30
  BMW     25  25  30
  Ferrari 35  28  35

21 / 33

Bayesian D-efficient designsMaximize information on "Main Effects" according to priors22 / 33

Bayesian D-efficient designs

Maximize information on "Main Effects" according to priors

Attribute	Levels	Prior
Brand	GM, BMW, Ferrari	0, 1, 2
Price	$20k, $40k, $100k	0, -1, -4

$v_j = 1 \delta^{\mathrm{BMW}} + 2 \delta^{\mathrm{Ferrari}} -1 \delta^{\mathrm{40k}} -4 \delta^{\mathrm{100k}}$

22 / 33

Bayesian D-efficient designsMaximize information on "Main Effects" according to priors

Attribute
Levels
Prior


Brand
GM, BMW, Ferrari
0, 1, 2

Price
$20k, $40k, $100k
0, -1, -4

Attribute counts:
brand:
  GM    BMW   Ferrari
  93    90     86
price:
  20k  40k 100k
  97   93   78
23 / 33

Bayesian D-efficient designsMaximize information on "Main Effects" according to priors

Attribute
Levels
Prior


Brand
GM, BMW, Ferrari
0, 1, 2

Price
$20k, $40k, $100k
0, -1, -4

Attribute counts:
brand:
  GM    BMW   Ferrari
  93    90     86
price:
  20k  40k 100k
  97   93   78
Pairwise attribute counts:
brand & price:
          20k 40k 100k
  GM      52  41  0
  BMW     30  30  30
  Ferrari 15  22  49
23 / 33

Negative of the hessian evaluated at a set of parameters is called the "Information Matrix"\boldsymbol{I}(\boldsymbol{\beta}) = - \nabla_{\boldsymbol{\beta}}^2 \ln (\mathcal{L})24 / 33

"D-optimal" designs attempt to minimize the
"D-error" of a design

$D = |\boldsymbol{I}(\boldsymbol{\beta})| ^{-1/p}$

where $p$ is the number of coefficients in the model

25 / 33

Finding Efficient DesignsOpen design-efficiency.qmd26 / 33

20:00

Your Turn

Individually, create a Bayesian D-efficient fractional factorial survey design. Inspect the attribute balance and overlap.
Compare your results with your teammates.

27 / 33

Week 10: DOE & Power Analysis1. Design of Experiment2. Design Efficiency3. Power Analysis28 / 33

How many respondents do I need?29 / 33

How many respondents do I need
to get X level of precision on \boldsymbol{\beta}?30 / 33

n <- seq(100)
se <- 1/sqrt(n)
plot(n, se, type = "l")

Standard errors also decrease with:

Fewer attributes
Fewer levels in each categorical attribute
More questions per respondent

31 / 33

Using {cbcTools}, we can run simulations to determine the necessary sample size for a specific modelOpen powerAnalysis.qmd32 / 33

20:00

Your Turn

Individually:

Using the survey design you created in the last practice, conduct a power analysis to determine the necessary sample size to achieve a 0.05 significance level on your parameter estimates.
Compare your results with your teammates.

33 / 33

Week 10: DOE & Power Analysis

EMSE 6035: Marketing Analytics for Design Decisions

John Paul Helveston

October 30, 2024

1 / 33

Quiz 4

10:00

Link is in the #class channel

2 / 33

Week 10: DOE & Power Analysis1. Design of Experiment2. Design Efficiency3. Power Analysis3 / 33

Week 10: DOE & Power Analysis1. Design of Experiment2. Design Efficiency3. Power Analysis4 / 33

Main & Interaction Effects5 / 33

Full design space for 3 effects: A, B, C

6 / 33

Full design space for 3 effects: A, B, C

Example: Cars

A: Electric? (Yes+ or No-)

B: Warranty? (Yes+ or No-)

C: Ford? (Yes+ or No-)

7 / 33

Main Effects

$ME(a) =$

$\left( \frac{A + AB + AC + ABC}{4}\right) -$

$\left( \frac{I + B + C + BC}{4}\right)$

(A: Electric? Yes+ or No-)

8 / 33

Interaction Effects

$INT(ab) =$

$\frac{1}{2}\left[ \left( \frac{AB + ABC}{2}\right) - \left( \frac{B + BC}{2}\right) \right] -$

$\frac{1}{2}\left[ \left( \frac{A + AC}{2}\right) - \left( \frac{I + C}{2}\right) \right]$

9 / 33

Example: Wine Pairings

meat
wine


fish
white 

fish
red

steak
white 

steak
red


10 / 33

meat	wine
fish	white
fish	red
steak	white
steak	red

Example: Wine Pairings

meat
wine


fish
white 

fish
red

steak
white 

steak
red


Main Effects
meat: Fish or Steak?
wine: Red or White?

10 / 33

meat	wine
fish	white
fish	red
steak	white
steak	red

Example: Wine Pairings

meat
wine


fish
white

fish
red

steak
white

steak
red


Main Effects
meat: Fish or Steak?
wine: Red or White?

Interaction Effects
meat*wine: Red or White wine with Steak?
meat*wine: Red or White wine with Fish?

11 / 33

meat	wine
fish	white
fish	red
steak	white
steak	red

Open interactions.qmd12 / 33

Fractional vs Full Factorial Designs13 / 33

Full Factorial DesignExample: Cars
A: Electric? (Yes+ or No-)
B: Warranty? (Yes+ or No-)
C: Ford? (Yes+ or No-)
library(cbcTools)
profiles <- cbc_profiles(
    electric = c(1, 0),
    warranty = c(1, 0),
    ford     = c(1, 0)
)
profiles

#>   profileID electric warranty ford
#> 1         1        1        1    1
#> 2         2        0        1    1
#> 3         3        1        0    1
#> 4         4        0        0    1
#> 5         5        1        1    0
#> 6         6        0        1    0
#> 7         7        1        0    0
#> 8         8        0        0    0
14 / 33

Full Factorial Design

Balanced?

All levels appear an equal number of times.

Orthogonal?

All pairs of levels appear together an equal number of times.

library(cbcTools)
profiles <- cbc_profiles(
    electric = c(1, 0),
    warranty = c(1, 0),
    ford     = c(1, 0)
)
profiles

#>   profileID electric warranty ford
#> 1         1        1        1    1
#> 2         2        0        1    1
#> 3         3        1        0    1
#> 4         4        0        0    1
#> 5         5        1        1    0
#> 6         6        0        1    0
#> 7         7        1        0    0
#> 8         8        0        0    0

15 / 33

Fractional Factorial Design

Balanced?

All levels appear an equal number of times.

Orthogonal?

All pairs of levels appear together an equal number of times.

profiles[c(1, 3, 5, 6),]

#>   profileID electric warranty ford
#> 1         1        1        1    1
#> 3         3        1        0    1
#> 5         5        1        1    0
#> 6         6        0        1    0

16 / 33

Comparing Full and Fractional Factorial DesignsOpen balance-orthogonality.qmd17 / 33

Practice Question 1

Consider the following experiment design

a	b	c	Effect
+	-	-	A
-	+	-	B
+	-	+	AC
-	+	+	BC

a) Is the design balanced? Is is orthogonal?

b) Write out the equation to compute the main effect for a, b, and c.

c) Are any main effects confounded? If so, what are they confounded with?

18 / 33

Week 10: DOE & Power Analysis1. Design of Experiment2. Design Efficiency3. Power Analysis19 / 33

A simple conjoint experiment about cars

Attribute	Levels
Brand	GM, BMW, Ferrari
Price	$20k, $40k, $100k

Design: 9 choice sets, 3 alternatives each

20 / 33

A simple conjoint experiment about cars

Attribute	Levels
Brand	GM, BMW, Ferrari
Price	$20k, $40k, $100k

Design: 9 choice sets, 3 alternatives each

Attribute counts:
brand:
  GM   BMW  Ferrari
  10    11    6
price:
 20k  40k 100k
  9    9   9

20 / 33

A simple conjoint experiment about cars

Attribute	Levels
Brand	GM, BMW, Ferrari
Price	$20k, $40k, $100k

Design: 9 choice sets, 3 alternatives each

Attribute counts:
brand:
  GM   BMW  Ferrari
  10    11    6
price:
 20k  40k 100k
  9    9   9

Pairwise attribute counts:
brand & price:
          20k 40k 100k
  GM        3   0    7
  BMW       4   5    2
  Ferrari   2   4    0

20 / 33

A simple conjoint experiment about cars

Attribute	Levels
Brand	GM, BMW, Ferrari
Price	$20k, $40k, $100k

Design: 90 choice sets, 3 alternatives each

21 / 33

A simple conjoint experiment about cars

Attribute	Levels
Brand	GM, BMW, Ferrari
Price	$20k, $40k, $100k

Design: 90 choice sets, 3 alternatives each

Attribute counts:
brand:
  GM    BMW   Ferrari
  92    80     98
price:
  20k  40k 100k
  91   84   95

21 / 33

A simple conjoint experiment about cars

Attribute	Levels
Brand	GM, BMW, Ferrari
Price	$20k, $40k, $100k

Design: 90 choice sets, 3 alternatives each

Attribute counts:
brand:
  GM    BMW   Ferrari
  92    80     98
price:
  20k  40k 100k
  91   84   95

Pairwise attribute counts:
brand & price:
          20k 40k 100k
  GM      31  31  30
  BMW     25  25  30
  Ferrari 35  28  35

21 / 33

Bayesian D-efficient designsMaximize information on "Main Effects" according to priors22 / 33

Bayesian D-efficient designs

Maximize information on "Main Effects" according to priors

Attribute	Levels	Prior
Brand	GM, BMW, Ferrari	0, 1, 2
Price	$20k, $40k, $100k	0, -1, -4

$v_j = 1 \delta^{\mathrm{BMW}} + 2 \delta^{\mathrm{Ferrari}} -1 \delta^{\mathrm{40k}} -4 \delta^{\mathrm{100k}}$

22 / 33

Bayesian D-efficient designsMaximize information on "Main Effects" according to priors

Attribute
Levels
Prior


Brand
GM, BMW, Ferrari
0, 1, 2

Price
$20k, $40k, $100k
0, -1, -4

Attribute counts:
brand:
  GM    BMW   Ferrari
  93    90     86
price:
  20k  40k 100k
  97   93   78
23 / 33

Bayesian D-efficient designsMaximize information on "Main Effects" according to priors

Attribute
Levels
Prior


Brand
GM, BMW, Ferrari
0, 1, 2

Price
$20k, $40k, $100k
0, -1, -4

Attribute counts:
brand:
  GM    BMW   Ferrari
  93    90     86
price:
  20k  40k 100k
  97   93   78
Pairwise attribute counts:
brand & price:
          20k 40k 100k
  GM      52  41  0
  BMW     30  30  30
  Ferrari 15  22  49
23 / 33

Negative of the hessian evaluated at a set of parameters is called the "Information Matrix"\boldsymbol{I}(\boldsymbol{\beta}) = - \nabla_{\boldsymbol{\beta}}^2 \ln (\mathcal{L})24 / 33

"D-optimal" designs attempt to minimize the
"D-error" of a design

$D = |\boldsymbol{I}(\boldsymbol{\beta})| ^{-1/p}$

where $p$ is the number of coefficients in the model

25 / 33

Finding Efficient DesignsOpen design-efficiency.qmd26 / 33

20:00

Your Turn

Individually, create a Bayesian D-efficient fractional factorial survey design. Inspect the attribute balance and overlap.
Compare your results with your teammates.

27 / 33

Week 10: DOE & Power Analysis1. Design of Experiment2. Design Efficiency3. Power Analysis28 / 33

How many respondents do I need?29 / 33

How many respondents do I need
to get X level of precision on \boldsymbol{\beta}?30 / 33

n <- seq(100)
se <- 1/sqrt(n)
plot(n, se, type = "l")

Standard errors also decrease with:

Fewer attributes
Fewer levels in each categorical attribute
More questions per respondent

31 / 33

Using {cbcTools}, we can run simulations to determine the necessary sample size for a specific modelOpen powerAnalysis.qmd32 / 33

20:00

Your Turn

Individually:

Using the survey design you created in the last practice, conduct a power analysis to determine the necessary sample size to achieve a 0.05 significance level on your parameter estimates.
Compare your results with your teammates.

33 / 33

↑, ←, Pg Up, k	Go to previous slide
↓, →, Pg Dn, Space, j	Go to next slide
Home	Go to first slide
End	Go to last slide
Number + Return	Go to specific slide
b / m / f	Toggle blackout / mirrored / fullscreen mode
c	Clone slideshow
p	Toggle presenter mode
t	Restart the presentation timer
?, h	Toggle this help
o	Tile View: Overview of Slides

Week 10: DOE & Power Analysis

EMSE 6035: Marketing Analytics for Design Decisions

John Paul Helveston

October 30, 2024

Quiz 4

Link is in the #class channel

Week 10: DOE & Power Analysis

1. Design of Experiment

2. Design Efficiency

3. Power Analysis

Week 10: DOE & Power Analysis

1. Design of Experiment

2. Design Efficiency

3. Power Analysis

Main & Interaction Effects

Full design space for 3 effects: A, B, C

Full design space for 3 effects: A, B, C

Example: Cars

A: Electric? (Yes+ or No-)

B: Warranty? (Yes+ or No-)

C: Ford? (Yes+ or No-)

Main Effects

Interaction Effects

Example: Wine Pairings

Example: Wine Pairings

Main Effects

Example: Wine Pairings

Main Effects

Interaction Effects

Open interactions.qmd

Fractional vs Full Factorial Designs

Full Factorial Design

Example: Cars

A: Electric? (Yes+ or No-)

B: Warranty? (Yes+ or No-)

C: Ford? (Yes+ or No-)

Full Factorial Design

Balanced?

Orthogonal?

Fractional Factorial Design

Balanced?

Orthogonal?

Comparing Full and Fractional Factorial Designs

Open balance-orthogonality.qmd

Practice Question 1

Week 10: DOE & Power Analysis

1. Design of Experiment

2. Design Efficiency

3. Power Analysis

A simple conjoint experiment about cars

A simple conjoint experiment about cars

A simple conjoint experiment about cars

A simple conjoint experiment about cars

A simple conjoint experiment about cars

A simple conjoint experiment about cars

Bayesian D-efficient designs

Maximize information on "Main Effects" according to priors

Bayesian D-efficient designs

Maximize information on "Main Effects" according to priors

Bayesian D-efficient designs

Maximize information on "Main Effects" according to priors

Bayesian D-efficient designs

Maximize information on "Main Effects" according to priors

Negative of the hessian evaluated at a set of parameters is called the "Information Matrix"

\boldsymbol{I}(\boldsymbol{\beta}) = - \nabla_{\boldsymbol{\beta}}^2 \ln (\mathcal{L})\boldsymbol{I}(\boldsymbol{\beta}) = - \nabla_{\boldsymbol{\beta}}^2 \ln (\mathcal{L})

"D-optimal" designs attempt to minimize the"D-error" of a design

D = |\boldsymbol{I}(\boldsymbol{\beta})| ^{-1/p}D = |\boldsymbol{I}(\boldsymbol{\beta})| ^{-1/p}

Finding Efficient Designs

Open design-efficiency.qmd

Your Turn

Week 10: DOE & Power Analysis

1. Design of Experiment

2. Design Efficiency

3. Power Analysis

How many respondents do I need?

How many respondents do I needto get X level of precision on \boldsymbol{\beta}\boldsymbol{\beta}?

Standard errors are inversely related to \sqrt{N}\sqrt{N}

Using {cbcTools}, we can run simulations to determine the necessary sample size for a specific model

Open powerAnalysis.qmd

Your Turn

Open `interactions.qmd`

Open `balance-orthogonality.qmd`

$\boldsymbol{I}(\boldsymbol{\beta}) = - \nabla_{\boldsymbol{\beta}}^2 \ln (\mathcal{L})$

"D-optimal" designs attempt to minimize the
"D-error" of a design

$D = |\boldsymbol{I}(\boldsymbol{\beta})| ^{-1/p}$

Open `design-efficiency.qmd`

How many respondents do I need
to get X level of precision on $\boldsymbol{\beta}$ ?

Standard errors are inversely related to $\sqrt{N}$

Open `powerAnalysis.qmd`

Open `interactions.qmd`

Open `balance-orthogonality.qmd`

$\boldsymbol{I}(\boldsymbol{\beta}) = - \nabla_{\boldsymbol{\beta}}^2 \ln (\mathcal{L})$

"D-optimal" designs attempt to minimize the
"D-error" of a design