Welcome to Psychology 504. This is a graduate course in research design in psychology. We will cover topics in the design and analysis that take you through the development of a research question to producing a research proposal. Topics covered include replicability, transparency, principles of causal inference, psychometrics, designing research procedures, developing analysis plans, and power calculations. We will also discuss ethical issues in research. This course is a “hands-on” class, meaning you’ll apply the concepts we study and discuss to your own research.
The final product of this course is twofold. First, you will produce a prospectus for your second year project or another project of your choosing. Second, and more importantly, the final product is you. You’ll become a more sophisticated researcher and psychologist as you develop your ability to think critically about research and ideas.
A research prospectus is a proposal. Specifically, a prospectus includes, at least, the following:
- The context of your research.
- A detailed and clear statement of the primary research question.
- A thorough description of your sample, where they come from and details about them. This includes meta-analyses (where the sample is a set of studies) as well as secondary data analyses (where the sample is already defined).
- Your planned research procedures. Experimental stimuli, treatments, measurement and assessment procedures, length of the study, and so on.
- A description and evaluation of your measures, including psychometric evidence (i.e., reliability and validity evidence).
- A power analysis, including a justification of your proposed effect size and how you settled on the particular size you chose.
- An analysis plan that describes how you are going to statistically analyze your data. Your specific model, including equations if relevant, and how your specific model maps onto your research question. You should describe what parameters from your model specifically address your research question, why your model is appropriate, and how you are going to implement it. You should address how you deal with missing data. You should also briefly describe what kinds of tables and figures you’ll use to present your data and results. In other words, you should describe how you’ll evaluate your results.
What we’re going to do
This class is a 15-week workshop to help you do the following:
- Learn key concepts in research design, including applying what you’ve learned in statistics courses to your project.
- Apply those concepts to your research project.
- Learn to present your ideas orally and in writing.
- Learn to to provide useful and actionable feedback (anyone can say more research is needed or you need a more generalizable sample).
- Get better at writing academically.
- Put in the work it takes to design a good study, whatever the form of the study.
We’ll weave lecture, discussion, and workshop activities. Some weeks will be heavier on the lecture and discussion and some weeks will be heavier on the workshop. Homework will include some reading and prepping for the workshop for that day. I expect you to get a lot of work done on your prospectus during the course of the semester as well as during each class period. You must be prepared for class each week for you to successfully make headway on your prospectus. I aim to keep the work load manageable each week so that you can be prepared. If you take advantage of this class, you’ll learn a ton (I mean it) and you’ll get a lot of work done on your project.
What are workshops?
I draw the term workshop from writing courses and groups. In a good writing course, you share material you’re working on to get feedback. This is called “workshopping”. In a creative writing course, you might work on “showing not telling” what your characters do. In the workshop, you bring portions of a story to class and work with your classmates to improve your execution of “showing not telling”. In a poetry workshop, you mightt work on the conceptual aspects of your poem or its meter and rhythm, and your classmates would provide feedback about how you can improve those aspects of the poem.
In this class, we’re going to workshop things like written statements of our primary research question, evaluations of our chosen measures, or power analyses. You’ll all be working on the same thing each class period, so you’ll see examples in addition to your own.
There are four primary benefits of workshopping material:
- Workshops keep you on task. If you aren’t prepared, you can’t participate.
- You get feedback. Your readers are your best source of feedback and you’ll get it frequently in a workshop.
- You learn to deal with and use feedback. Many of us are perfectionists and critical feedback stings. It can make us feel like we aren’t good or can’t succeed. The only way to get over this is to get feedback and then get more (by more, I mean lots). Some feedback will be useful and some won’t be–but your thinking and writing will be clearer because of it. I believe some of my best writing has come about due to feedback that wasn’t quite on target and I had to explain why the feedback was off course.
- You learn to give feedback. Some of us are shy, some are rude, and some are too nice. Workshops, in order to be useful, require that you give constructive feedback. I’m committed to you learning to give constructive feedback. If your classmate has a paragraph that wanders and doesn’t support their primary research question, tell them. Be nice about it, but be frank.
Finally, though the primary inspiration for using the term “workshop” is writing groups, I also think of this class like a 15-week professional development workshop. In other words, this class is like a longer version of what you might do at a conference to learn the basics of Acceptance and Commitment Therapy or Item Response Theory. These workshops typically have didactic and experiential portions. We will too.
The course schedule can be found on Learning Suite. The schedule includes the reading assignments and due dates for all assignments.
Grading of Workshops and Feedback
Each workshop results in a section of your prospectus that you turn into me. 70% of the grade is based on how well you meet requirements of that section (see the workshop descriptions for the requirements) and 30% is based on the quality of your writing (writing should be clear, concise, free of spelling and grammar errors, etc.). At the end of the course, you will turn in a completed paper, along with a summary of the improvements you have made (similar to a cover letter in the peer review process). The rubric for the final project is the same as each part. However, for the final portion you also need to show that you have responded to the feedback you’ve received (from me and colleagues).
Each time you workshop material in class, you’ll also give written feedback to colleagues. The workshop descriptions provide details about the types of things I want you to think about and provide feedback on. These are only suggestions and I expect you to be thoughtful. You’re expected to give actionable feedback and you’re expected to be respectful and thoughtful. 70% of your grade for the feedback is based on the quality of your feedback and 30% on your tone and writing clarity. I will provide examples to help you.
You’ll present your prospectus at the end of the semester. Details can be found here:
There will be one exam about halfway through the semester.
The Learning Outcomes from this course can be found here.
The university and course policies can be found on Learning Suite.