Unit 08

Unit 8:
How to Evaluate Scientific Reliability and Validity
 IMPORTANT: Unit 8: Assignment #6 will require advance planning!
Unit 8: Assignment #1 (due before 11:59 pm Central on THU JUL 11):

  1. To learn what reliability and validity mean in research:
    1. Read an excerpt from Sommer’s (no date) article, “Introduction: Reliability and Validity.”
    2. Read a student from Bangor University’s (2011) blog post, “Reliability vs. Validity.”
    3. To make sure you understand the difference between reliability and validity, read to understand Professor Gernsbacher’s handout, “Illustrating Reliability versus Validity of a Measurement.”
  2. To learn what external versus internal validity means in research, read Bradley’s (no date), PPT, “Reliability, Validity, and Bias.”
    1. Although you should be sure you understand both terms — reliability and validity — and you should understand their difference, this assignment will focus on reliability. (Another assignment will focus on validity.)
    2. You’ve probably noticed that reliability is, in many ways, the same as replicability and reproducibility: Reliable findings can be replicated and reproduced.
    3. Therefore, the issues you learned about replicability and reproducibility in Unit 4 (particularly, Unit 4: Assignments #5 and #6) are issues of reliability.
  3. To dig even deeper into the issues of reliability, replicability, and reproducibility, read Munafó et al.’s (2017) article, “A Manifesto for Reproducible Science.”
    1. NOTE: Munafó et al.’s (2017) article is complex (it’s not Huffington Post-style writing).
      • You might need to read it a few times to understand it. Please be persistent.
    2. Munafó et al. (2017) suggest 10 procedures for making science, including psychological science, more reproducible (replicable and reliable).
      • Their 10 suggested procedures are listed in Table 1 on the third page, and each is explained in the article.
    3. Choose three of Munafó et al.’s (2017) 10 suggested procedures.
      • Choose the three suggested procedures that you like the most (or that you understand the best).
    4. Write one five-paragraph Examples-Style essay to support the thesis “Scientists Can Improve Research Reliability.”
      • For this essay, the three procedures you have chosen from Munafó et al.’s (2017) article will be your three ‘examples’ in your Examples-Style essay.
  4. If you need to review how to write a Five-Paragraph Essay, including how to write an Examples-Style essay, watch again Professor Gernsbacher’s lecture video, “The Five-Paragraph Model” (a transcript of the video is available here).
    1. First, check your essay to make sure your Introduction Paragraph has a hook and a Thesis Statement.
    2. Second, check your Thesis Statement to make sure that it is one sentence that incorporates your three examples/procedures.
    3. Third, check your essay to make sure it has three Supporting Paragraphs, each introducing one of your three examples/procedures.
    4. Fourth, check your essay to make sure it has a Conclusion Paragraph.
    5. Fifth, check your Conclusion Paragraph to make sure it has one sentence that restates your Thesis Statement (summarizing your three examples/procedures).
    6. Sixth, check ALL five of your paragraphs to make sure each paragraph has five sentences: a Topic Sentence, three Supporting Sentences, and a Conclusion Sentence.
  5. Because you’ll be drawing a lot of ideas from Munafó et al.’s (2017) article, review Unit 6: Assignment #4 to make sure you don’t plagiarize.
    1. Try to not make Munafó and his co-authors the grammatical subjects of your sentences. Instead, make the ideas (e.g., the three examples/procedures you’ve chosen) the grammatical subjects of your sentences.
    2. Remember, from Unit 6, how to synthesize psychological science, not Mad Lib it. Be sure to review the Synthesizing Research cheat-sheet.
    3. Be sure to include in-text citations to Munafó et al.’s (2017) article whenever needed, which is — of course — after each and every sentence in which you paraphrase any idea from their article (even if you are paraphrasing an idea from the same article as you paraphrased in a previous sentence).
    4. Remember that every sentence you write that does not contain only your own words or your own ideas needs to have attribution with an in-text citation (i.e., your readers need to be able to trace words or ideas that are not your own to their original source).
    5. Also be sure to include a full citation to Munafó et al.’s (2017) article, in APA style, at the end of your essay.
    6. Save your essay as a PDF and name the file YourLastname_PSY-225_ReliabilityEssay.pdf.
  6. Go to the Unit 8: Assignment #1 Discussion Board and attach your essay, saved as a PDF.
    • Remember to use the “Attach” too and not to use the “Files” tool or the “Documents” tool.

Unit 8: Assignment #2 (due before 11:59 pm Central on THU JUL 11):

  1. Reliable and valid studies have reliable and valid operational definitions. To learn what an operational definition is, do the following:
    1. First, read Bluestone’s (no date) PPT slide, “Operational Definitions.”
      • When reading Bluestone’s PPT, make sure you understand that “An operational definition is how we (the researchers) decide to measure our variables in our study (variable = anything that can be measured).” And
      • “There are usually hundreds of ways to measure a variable (e.g., a behavior).”
    2. Second, read this (no date) definition from Quality Advisor℠.
      • When reading Quality Advisor’s definition, make sure you understand that “An operational definition … is a clear, concise detailed definition of a measure.”
  2. To understand why consistent, precise, logical, and justifiable operational definitions are crucial for reliability and validity:
    1. Read Dewey’s (2007) chapter, “Operational Definitions.”
      • When reading Dewey’s chapter, make sure you understand that “operational definitions … tell how to measure or detect something.”
  3. To appreciate the difficulty of generating reliable, precise, and valid operational definitions,
    1. Read this excerpt from Lammers and Badia
’s (2013) textbook, Fundamentals of Behavioral Research.
      • When reading the excerpt from Lammers and Badia’s chapter, make sure you understand that “An essential component of an operational definition is measurement.”
      • At this point, you should have a good understanding of what it means to operationally define a variable; it means specifying (defining) the variable’s measurement.
  4. To demonstrate the difficulty of generating reliable, precise, and valid operational definitions, complete this fillable PDF.
    1. First, download the unfilled PDF and save it on your own laptop or tablet.
      • Rename the unfilled PDF to be YourLastName_PSY-225_Gernsbacher_OperationalDefinitions_Fillable.pdf. (In other words, add your LastName to the beginning of the filename.)
      • On your computer, open a PDF writer app, such as Preview, Adobe Reader, or the like.
      • Be sure to open your PDF writer app before you open the unfilled but re-named PDF from your laptop/tablet.
      • From within your PDF writer app, open the unfilled PDF, which you have already saved onto your laptop or tablet and re-named.
    2. Second, using your PDF writer app, type (rather than handwrite) into the PDF with your operational definition of each of the bolded terms.
      • The PDF contains a list of 10 fortunes, such as one would see in a Magic 8 Ball game.
      • Each fortune has at least one term bolded.
      • Your task is to provide an operational definition for each bolded term.
      • For example, if the fortune is “You will inherit some money or a small piece of land,” you need to fill in the PDF with your operational definition of “some money” and “a small piece of land.”
        • What is your operational definition of “some money“? Is it $50? $100? $1000? $100,000? One million dollars? Five million dollars?
          • Whatever your operational definition of “some money” is, type into the PDF with that measurement.
        • What is your operational definition of “a small piece of land“? Is it the size of a typical front yard? The size of a football field? An acre? Several acres? A hundred acres?
          • Whatever your operational definition of “a small piece of land” is, type into the PDF with that measurement.
      • As Lammers and Badia
 write in their (2013) textbook, “there are no correct answers to these questions. However, the researcher must decide what is going to be meant by each of the variables in a particular study.” And that is what you will be doing in this assignment!
      • OPTIONAL: For a real-world example of the difficulty in operationally defining variables, see this article that reports how 25,000 US adults operationally defined “rich.”
    3. Third, teach a friend (or family member) what an operational definition is.
      • Then, ask this friend/family member to tell you their operational definition of each of the bolded terms in the fillable PDF.
      • Type into the PDF your friend/family member’s operational definitions.
      • Although you will first need to explain to your friend/family member what an operational definition is, do not tell your friend/family member your operational definitions of the bolded terms before your friend/family member tells you their operational definitions
    4. Fourth, teach a second friend (or family member) what an operational definition is.
      • Then, ask this second friend/family member to tell you their operational definition of each of the bolded terms.
      • Type into the PDF your second friend/family member’s operational definitions.
      • Again, you will need to explain to your friend/family member what an operational definition is, BUT do not tell your second friend/family member your operational definitions of the bolded terms (or your first friend/family member’s operational definitions) before your second friend/family member tells you their operational definitions.
  5. Go to the Unit 8: Assignment #2 Discussion Board and make a new post of at least 200 words in which you do the following:
    1. First, identify which of the bolded terms you and your friends/family members showed the most AND showed the least RELIABILITY (consistency) on when each of you (independently) defined those terms operationally.
      • Speculate why those bolded terms led to the most and the least reliability (consistency).
    2. Second, identify which of the bolded terms you and your friends/family members operationally defined in the most and operationally defined in the least VALID (credible and believable) ways.
      • Speculate why those bolded terms led to the most and the least validity (credibility).
    3. Third, attach (not embed, but attach) your filled-in PDF.

Unit 8: Assignment #3 (due before 11:59 pm Central on FRI JUL 12):

  1. Although the construct of validity is a general research methods term, when the construct of validity is applied to tests, such as IQ tests or personality inventories, we use the specific term “test validity.”
  2. To learn what the specific term “test validity” means and to be introduced to the major ways that test validity can be assessed, read a section from Heffner’s (no date) “Chapter 7.3 Test Validity.”
  3. Now, think back to one of the exams you took last semester in one of your courses. In fact, think about the most memorable exam you took (either because it was memorably positive or memorably negative). Refer to this exam as “Text X” (for purposes of this assignment).
  4. Go to the Unit 8: Assignment #3 Discussion Board and make a new post of at least 200 words in which you discuss Test X’s
    1. construct validity (did the exam measure what it was supposed to measure?);
    2. concurrent validity (was the exam like other comparable exams, for example, the exams your friends took in a similar course?);
    3. content validity (were the items on the test representative of the content you were expected to learn?); and
    4. predictive validity (how well did the exam predict your ability to use the information in the future?).

Unit 8: Assignment #4 (due before 11:59 pm Central on FRI JUL 12):

  1. Because we’re a bit more than halfway through the course, it’s a good time to revisit things you’ve learned so far in the course (which is why traditional lecture-based courses usually have midterm exams around the middle of the term).
  2. Listed below are Assignments you previously completed (or were supposed to complete). Choose one Assignment from the list that you enjoyed doing before and re-do that Assignment, according to the Assignment’s original instructions AND the additional requirements specified below.
    1. Unit 1: Assignment #3 (choose three situations that are different from the three situations you chose before and are different from the three situations that other students chose before)
    2. Unit 1: Assignment #5 (teach the Cognitive Biases to three people who are different from the three people you taught before)
    3. Unit 2: Assignment #5 (make a different teaching document than you made before, aimed at a different audience than you aimed at before, and using a different medium than you used before)
    4. Unit 3: Assignment #4 (write your two essays on two quotes that are different from the two quotes you wrote essays on before; however, as before, one of your two essays must be based on a “Reasons/Arguments Prompt” quote and the other must be based on an “Examples Prompt” quote)
    5. Unit 3: Assignment #5 (write your two essays in response to two tasks that are different from the two tasks you completed before)
    6. Unit 4: Assignment #2 (find and analyze three different news stories than you found before)
    7. Unit 4: Assignment #3 (teach correlation is not causation to three people who are different from the three people you taught before)
    8. Unit 5: Assignment #1 (choose three topics that are different from the topics you chose; therefore, you’ll need to find three articles that are different from the articles you found before)
    9. Unit 5: Assignment #2 (find and analyze ten popular press articles that are different from the ten popular press articles you found before)
    10. Unit 7: Assignment #2 (make a different teaching document than you made before, aimed at a different audience than you aimed at before, and using a different medium than you used before)
    11. Unit 7: Assignment #3 (choose a different set of three of your 9 articles than you analyzed before)
  3. Go to the Unit 8: Assignment #4 Discussion Board and make a new post containing everything that was previously required for the Assignment that you’ve now re-done.
    1. Also, explain in just a sentence or two why you chose this Assignment to do again.
    2. Be sure to state somewhere in your post the Unit number and Assignment number (e.g., Unit 1: Assignment #3) of the Assignment you chose to do again.

Unit 8: Assignment #5 (due before 11:59 pm Central on SUN JUL 14):

  1. Successful learning also requires revisiting previous challenges and overcoming those challenges.
  2. From the Assignments listed above, in Unit 8: Assignment #4, select one Assignment that you previously completed (or were supposed to complete), but you struggled (a bit or a lot) when completing it.
    1. Re-do that Assignment, according to the Assignment’s original instructions AND the additional requirements specified in the list in Unit 8: Assignment #4.
    2. One further constraint is that the Assignment you choose to re-do for Unit 8: Assignment #5 needs to be different from the Assignment you chose to re-do for Unit 8: Assignment #4.
  3. Go to the Unit 8: Assignment #5 Discussion Board and make a new post in which you post everything that was previously required for the assignment that you’ve now re-done.
    1. Also, explain in just a sentence or two why you chose this Assignment to do again.
    2. Be sure to state somewhere in your post the Unit number and Assignment number (e.g., Unit 1: Assignment #3) of the Assignment you chose to do again.

Unit 8: Assignment #6 (due before 11:59 pm Central on SUN JUL 14):

  1. At this point in the course, we will construct new Chat Groups. Please follow closely the instructions below.
    • IMPORTANT: If the due date for this assignment is approaching, and you haven’t been contacted about being in a new Chat Group, reach out to ALL the students in your Section who are responsible for recruiting new Chat Groups
    • You should also monitor the Unit 8: Assignment #6 Discussion Board so that you will know which new Chat Groups have already been formed.
  2. If your last name is LAST alphabetically in your “old” Chat Group, these are your responsibilities for forming a new Chat Group:
    1. You’re responsible for recruiting new students to your new Chat Group. You can contact students via their wisc.edu email (which you can find on the “old” Chat Group list by clicking, or right-clicking, on any student’s name).
    2. You may recruit any student to your new Chat Group from your Section (but only from your Section) except
      • you may NOT recruit a student who was a member of your “old” Chat Group (because the goal during the second half of the course is to mix up the Chat Groups a bit); and
      • you may NOT recruit a student whose last name is also last alphabetically in their “old” Chat Group (because those students whose last names are last alphabetically in their old Chat Groups will also be responsible for recruiting students to their new Chat Group).
      • If you can’t remember what Section you are in, you might want to return to Unit 1: Assignment #2 (and the part of the assignment that said “From the Course How To, learn how to find out which section you are in and find out which section you are in”).
    3. If your “old” Chat Group had three members (including you), then your new Chat Group should also have three members (including you); if your “old” Chat Group had only two members, then your new Chat Group will have only two members (including you).
    4. When recruiting new members to your new Chat Group:
      • It does NOT matter if the students you are recruiting were previously in a two-member or three-member “old” Chat Group.
      • It only matters how many members you (the person whose last name comes LAST alphabetically and therefore the person doing the recruiting) had in your “old” Chat Group.
      • Similarly, it does NOT matter if the students you are recruiting previously worked together in their “old” Chat Group; it only matters that none of the students you are recruiting are students YOU worked with before in your “old” Chat Group.
    5. Appoint one member of your new Chat Group to set up the new Group Chat Space.
      • Remind this person that the instructions for setting up a new Group Chat Space are below in c.
      • You must NOT be the Chat Group member to set up the new Group Chat Space; rather, you must appoint someone else — a member of your new Chat Group — to set up the new Chat Group Space.
    6. Arrange with your new Chat Group a day and date for a one-hour time period when all of you can meet online to hold your Group Chat for the next Unit (Unit 9: Assignment #6).
    7. Decide which Chat Group member will be responsible for using the browser Chrome on a laptop (not a mobile device) during your Group Chat.
    8. Go to the Unit 8: Assignment #6 Discussion Board and make a new post in which you list the following:
      • the full name (e.g., The Reliables: Group X) of your new Chat Group (which will be the name of your “old” Chat Group);
      • the first and last name of the members of your new Chat Group (including you);
      • the first and last name of the student you appointed to create the new Group Chat Space;
      • the first and last name of the student who will be responsible for using the browser Chrome on a laptop (not a mobile device) during your Group Chat; and
      • the day (e.g., MON, TUE, WED), the calendar date (e.g., JUL 18, JUL 19, JUL 20), and the hour-long timeslot (e.g., 2:30 pm – 3:30 pm) that your new Chat Group has agreed to meet to hold your Group Chat for the next Unit (Unit 9: Assignment #6).
  3. If you are appointed to set up the new Chat Group Space for your new Chat Group, these are your responsibilities:
    1. Follow the instructions in the Course How To for “How To Set Up a Group Chat Space on Your Laptop” OR “How To Set Up a Group Chat Space on Your Mobile Device.”
    2. Title the Chat Space the name of the old Chat Group Space of the student who recruited you to the new Chat Group, AND add an underscore and a 2 to the Chat Space name to indicate the second half of the term (e.g., TheReliables_GroupX_SU24_2).
      • You are responsible for finding out the name of the old Chat Group Space of the student who recruited you to the new Chat Group.
    3. Take (and save) a screenshot of the name of your new Chat Group space, which shows who you added to the Chat Space. (If you don’t know how to take a screenshot, this website will help you.)
    4. Arrange with your new Chat Group a date and a one-hour time period when all of you can meet online to hold your Group Chat for the next Unit (Unit 9: Assignment #6).
    5. Decide which Chat Group member will be responsible for using the browser Chrome on a laptop (not a mobile device) during your Group Chat.
    6. Go to the Unit 8: Assignment #6 Discussion Board and make a post in which you
      • embed (not attach, but embed) the screenshot of your new Chat Group’s Chat Space; remember from the Course How To that you might need to size your image, and
      • list the day (e.g., MON, TUE, WED), the calendar date (e.g., JUL 18, JUL 19, JUL 20), and the hour-long timeslot (e.g, 2:30 – 3:30 pm) that your new Chat Group has agreed to meet to hold your Group Chat for the next Unit (Unit 9: Assignment #6).
  4. If you are in a NEW THREE-student Chat Group AND you were NOT appointed to set up the new Group Chat Space these are your responsibilities:
    1. Arrange with your new Chat Group a time when all of you can meet online for one hour to hold your Group Chat for the next Unit (Unit 9: Assignment #5).
    2. Decide which Chat Group member will be responsible for using the browser Chrome on a laptop (not a mobile device) during your Group Chat
    3. Go to the Unit 8: Assignment #6 Discussion Board and make a new post in which you list
      • the full name (e.g., The Reliables: Group X) of your new Chat Group (which will be the name of the “old” Chat Group of the student who recruited you to the new Chat Group);
      • the first and last names of all the members of your new Chat Group (including you);
      • the first and last name of the student who recruited you to the new Chat Group;
      • the first and last name of the student who will be responsible for using the browser Chrome on a laptop (not a mobile device) during your Group Chat; and
      • the day (e.g., MON, TUE, WED), the calendar date (e.g., JUL 18, JUL 19, JUL 20), and the hour-long timeslot (e.g, 2:30 – 3:30 pm) that your new Chat Group has agreed to meet to hold your Group Chat for the next Unit (Unit 9: Assignment #6).
  5. EVERYONE: Continuing our theme of re-visiting at the middle of the term what you’ve learned so far in the course:
    1. First, ensure that your Course Journal is up-to-date and contains entries for Units 1 through 8, as well as your Goals Setting Entry.
    2. Second, save your Course Journal, along with your Goals Setting Entry as a PDF, titled LastName_PSY-225_CourseJournal_1-8.pdf.
      • If your Course Journal is hand-written (or hand-drawn), you will need to scan the pages into a PDF.
    3. Third, go to the Unit 8: Assignment #6 Discussion Board and make a new post to which you will attach your Course Journal PDF.

Congratulations, you have finished Unit 8! Onward to Unit 9!