Pilot Analysis

Due: Nov 06 by 11:59pm

Weight: This assignment is worth 9% of your final grade.

Assessment: Your submission will be assessed using the rubric at the bottom of this page.

As a team, analyze your pilot survey data, and write a report of your findings. The course instructor will review and grade your analysis and report and provide feedback. The primary purpose of the report is to 1) lay some initial groundwork for analyzing your survey data (you will be able to re-use this code for your final analysis), and 2) gain insights to improve your final survey.

Your report should include three main components:

Follow the guidelines below to prepare your analysis and report.

1. Report Guidelines

Download and unzip this template for your report. You should write your report in the report.Rmd.Rmd` file. The template comes with some text and code explaining how to use it - you should delete this code & text as it is only for explanatory purposes. Be sure to adjust the content in the YAML:

  • Write your project title in the title field (and provide a subtitle if you wish, or delete the subtitle field).
  • In the author field, list the names of all teammates, e.g. author: Luke Skywalker, Leia Organa, and Han Solo.

Your report should be written in a clear, concise, and logically-structured manner so that a reader can easily understand the ideas presented. The report should be proof-read before submission to correct any grammatical or spelling errors. Spell checking can be automated by installing the spelling package and running this in the R console:

spelling::spell_check_files("report.Rmd")

All figures, tables, and equations should be referenced in the text.

2. Pilot Data Analysis

Before writing your report, you should first analyze your pilot data by following these steps:

  1. Download the data files from formr.org for your surveys and put them in the “data” folder in the report template.
  2. Open the clean_data.R file and write code to clean your pilot data and prepare it for modeling. Your cleaned choice data should be in a “long” format similar to the example choice data here.
  3. Save your cleaned data file as a .csv file in the “data” folder.
  4. Open the modeling.R file and write code to estimate preliminary models using your cleaned choice data.
  5. Open the power_analysis.R file and write code to conduct a power analysis to inform the sample size requirements for your final survey.

3. Pilot Analysis Report

Your report should contain the following items listed as separate sections.

NOTE: You may re-use content from earlier project assignments in this report, such as introductory paragraphs, images, etc.

Abstract

A few sentences summarizing the project and what you found from your pilot analysis. Include:

  1. What is the product / policy you are studying?
  2. What were the product attributes and design decisions you were interested in?
  3. At least one key insight you gained from your pilot analysis (e.g., a modeling result, a design change you will make for your final survey, etc.).

Introduction

Provide a brief description and background of the product / policy that motivates the study of it. Include a picture or diagram of the product. You should also discuss the key product attributes and decisions variables you chose to study in your pilot survey. This section can be largely similar to your proposal introduction.

Survey Design

This section should discuss the design choices you made in creating your pilot survey. Include the following:

  • Describe the eligibility requirements you chose and why you chose them.
  • What information about your respondents did you collect (e.g. critical information questions, demographics, etc.)?
  • Describe any educational material presented to the respondents.
  • Define and justify the attributes and levels chosen for your conjoint choice questions. Indicate the number of alternatives per choice question and the number of choice questions per respondent.
  • If you included a “no-choice” option, define its meaning and state how that was communicated to the respondent.
  • Include an example figure of a random conjoint question (this can be a screen shot).

You should include a full copy of your survey in an appendix (see the Appendix section below)

Pilot Data Analysis

This section should describe the analysis you carried out using your pilot data, including the following subsections:

Sample Description

Describe your respondent sample, including the total sample size, the total number of choice question responses, and summary statistics of how your sample varies across critical information questions, demographics, etc. You should also include a description of what your target demographics should look like and compare these to your pilot sample (obviously your pilot sample was your classmates, so it may not be too representative of the target).

Data Cleaning

Describe any filtering process you applied to your data. Summarize how many acceptable vs. rejected responses you kept in your analysis and justify how you made that determination.

Modeling

Write out the specific utility model you estimated. This online LaTeX equation editor might come in handy for this (also, some example LaTeX equations are provided in the solutions.Rmd files from previous classes). Include a summary table of your utility coefficient estimates with standard errors. You should also include summary figures of the estimated coefficients for all attributes. Your figures should include uncertainty bounds. Interpret and discuss the implications of your findings.

Power Analysis

You should also include a description of conclusions from your power analysis, including a summary figure of your power analysis results (standard errors versus sample size).

Changes to Final Survey

Based on your analyses, write a description of any changes you plan on making to your final survey (which should be based on your pilot survey). Consider the following:

  • Did you make any changes to the product attributes and / or levels?
  • Do you expect to be able to obtain the necessary sample size for your analysis?
  • Other useful feedback did you receive from respondents / classmates that informed a design change.

Appendix

You should include a full copy of your conjoint survey in an appendix (i.e., the same information and questions respondents saw when taking the survey). This can be the same content used in your Rmd files from your survey plan (updated to match what you actually included in your pilot survey). For the conjoint choice questions, include just one example since the specific levels were different for each choice question.

4. Knit and submit

Click the “knit” button to compile your report.Rmd file into a html web page, then create a zip file of everything in your pilot analysis folder. Name your file pilot-analysis.zip, then go to your team Box folder and submit your zip file in the “submissions” folder. Only one person from your team should submit the report.

Grading Rubric

35 Total Points

Category Excellent Good Needs work
Organization & Formatting 5
All formatting guidelines are followed.		 4
Most formatting guidelines are followed; only one or two components from the template are missing.		 3
Most formatting guidelines are followed; several components from the template are missing or incomplete.
Pilot Analysis 10 / 9
All analysis guidelines are followed, including data cleaning, modeling, and a power analysis. Code in related files runs without error and results can be fully reproduced.		 8 / 7 / 6
Most analysis guidelines are followed, including data cleaning, modeling, and a power analysis. Code in related files might have a few errors, but the results can still be reproduced with a few fixes.		 5 / 4 / 3
Some of the analysis guidelines components are incomplete or missing, including data cleaning, modeling, and a power analysis. Code in related files has multiple significant errors and results cannot be reproduced, even with efforts to correct errors.
Report Contents 15 / 14
All report guidelines are followed, including descriptions of the survey design, the pilot analysis results, and changes that will be made to the final survey.		 13 / 12 / 11
Most report guidelines are followed, including descriptions of the survey design, the pilot analysis results, and changes that will be made to the final survey. Some may be unclear or missing important details.		 10 / 9 / 8 / 7
Several report guidelines are not followed, including descriptions of the survey design, the pilot analysis results, and changes that will be made to the final survey. Some components are incomplete or missing.
Appendix 5
Appendix of the full survey is included.		 4
Appendix of the full survey is included, but some parts of the survey may be missing.		 3
No appendix of survey included