Unit 08

Unit 8:
How to Evaluate Scientific Reliability and Validity
Unit 8: Assignment #1 (due before 11:59 pm Central on MON JUL 16):

  1. To learn what reliability and validity mean in research:
    1. Read an excerpt from Sommer’s (no date) article, “Introduction: Reliability and Validity.”
    2. Read a student from Bangor University’s (2011) blog post, “Reliability vs. Validity.”
  2. To learn what external versus internal validity means in research, read Bradley’s (no date), PPT, “Reliability, Validity, and Bias.”
    1. Although you should be sure you understand both terms, reliability and validity, and their difference, this assignment will focus on reliability. (Another assignment will focus on validity.)
    2. You’ve probably noticed that reliability is, in many ways, the same as replicability and reproducibility: Reliable findings can be replicated and reproduced.
    3. Therefore, the issues you learned about replicability and reproducibility in Unit 4 (particularly, Unit 4: Assignments #5 and #6) are issues of reliability.
  3. To dig even deeper into the issues of reliability, replicability, and reproducibility, read Munafó et al.’s (2017) article, “A Manifesto for Reproducible Science.”
    1. NOTE: Munafó et al.’s (2017) article is complex (it’s not Huffington Post-style writing).
      • You might need to read it a few times to understand it. Please be persistent.
    2. Munafó et al. (2017) suggest 10 procedures for making science, including psychological science, more reproducible (replicable and reliable).
      • Their 10 suggested procedures are listed in Table 1 on the third page, and each is explained in the article.
    3. Choose three of Munafó et al.’s (2017) 10 suggested procedures.
      • Choose the three suggested procedures that you like the most (or that you understand the best).
    4. Write one five-paragraph Examples-Style essay to support the thesis “Scientists Can Improve Research Reliability.”
      • For this essay, the three procedures you have chosen from Munafó et al.’s (2017) article will be your three ‘examples’ in your Examples-Style essay.
  4. If you need to review how to write a Five-Paragraph Essay, including how to write an Examples-Style essay, watch again Professor Gernsbacher’s lecture video, “The Five-Paragraph Model” (a transcript of the video is available here).
    1. Check your essay to make sure your Introduction Paragraph has a hook and a Thesis Statement.
    2. Check your Thesis Statement to make sure that it summarizes your three examples/procedures.
    3. Check your essay to make sure it has three Supporting Paragraphs.
    4. Check each of your three Supporting Paragraphs to make sure each one has a Topic Sentence, three Supporting Sentences, and a Conclusion Sentence.
    5. Check your essay to make sure it has a Conclusion Paragraph.
    6. Check your Conclusion Paragraph to make sure it has a sentence that restates your Thesis Statement (summarizing your three examples/procedures).
  5. Because you’ll be drawing a lot of ideas from Munafó et al.’s (2017) article, review Unit 6: Assignment #4 to make sure you don’t plagiarize.
    1. Be sure to include in-text citations to Munafó et al.’s (2017) article when needed, but try to not make Munafó and his co-authors the subjects of your sentences.
    2. Make your ideas (e.g., the three procedures you’ve chosen) the subjects of your sentences.
    3. Remember, from Unit 6, how to synthesize psychological science, not Mad Lib it.
    4. Include a full citation to Munafó et al.’s (2017) article, in APA style, at the end of your essay.
    5. Save your essay as a PDF and name the file YourLastname_ReliabilityEssay.pdf.
  6. Go to the Unit 8: Assignment #1 Discussion Board and attach your essay, saved as a PDF.

Unit 8: Assignment #2 (due before 11:59 pm Central on MON JUL 16):

  1. Reliable and valid studies have reliable and valid operational definitions. To learn what operational definitions are, read Bluestone’s (no date) PPT slide, “Operational Definitions,” and read this (no date) definition from Quality Advisor℠.
  2. To understand why consistent, precise, logical, and justifiable operational definitions are crucial for reliability and validity, read Dewey’s (2007) chapter, “Operational Definitions.”
  3. To appreciate the difficulty of generating reliable, precise, and valid operational definitions, read this excerpt from Lammers and Badia
’s (2013) textbook, Fundamentals of Behavioral Research.
  4. To demonstrate the difficulty of generating reliable, precise, and valid operational definitions, complete this fillable PDF, which contains a list of 10 fortunes, such as one would see in a Magic 8 Ball game. Each fortune has at least one term bolded.
    1. First, download the unfilled PDF and save it on your own computer.
      • Rename the unfilled PDF to be YourLastName_PSY-225_Gernsbacher_OperationalDefinitions _Fillable.pdf. (In other words, add your LastName to the beginning of the filename.)
      • On your computer, open a PDF writer app, such as Preview, Adobe Reader, or the like.
      • Be sure to open your PDF writer app before you open the unfilled PDF from your computer.
      • From within your PDF writer app, open the unfilled PDF, which you have already saved onto your computer and re-named.
    2. Second, using your PDF writer app, fill in the PDF with your operational definition of each of the bolded terms.
      • For example, if the fortune is “You will inherit some money or a small piece of land,” fill in the PDF with your operational definition of “some money” (What are the criteria for “some money”? How will you distinguish “some money” from money that’s not “some money”?) and with your operational definition of “a small piece of land” (What are the criteria for “a small piece of land”? How will you distinguish “a small piece of land” from a large piece of land or no land at all?).
      • If you’re unclear what an operational definition is, it would be good to go back and read the articles in a., b., and c., above.
    3. Third, teach a friend (or family member) what operational definitions are.
      • Then, ask this friend/family member to tell you their operational definition of each of the bolded terms.
      • Fill in your friend/family member’s operational definitions on your PDF.
      • Although you will first need to explain to your friend/family member what an operational definition is, do not tell your friend/family member your operational definitions of the bolded terms before your friend/family member tells you their operational definitions
    4. Fourth, teach a second friend (or family member) what operational definitions are.
      • Then, ask this second friend/family member to tell you their operational definition of each of the bolded terms.
      • Fill in your second friend/family member’s operational definitions on your PDF.
      • Again, you will need to explain to your friend/family member what an operational definition is, BUT do not tell your second friend/family member your operational definitions of the bolded terms (or your first friend/family member’s operational definitions) before your second friend/family member tells you their operational definitions.
  5. Go to the Unit 8: Assignment #2 Discussion Board and make a new post of at least 200 words in which you do the following:
    1. Identify which of the bolded terms you and your friends/family members showed the most and the least reliability on when each of you (independently) defined those terms operationally.
      • Speculate why those terms led to the most versus least reliability.
    2. Identify which of the bolded terms you and your friends/family members operationally defined in the most versus least valid ways.
      • Speculate why those terms led to the most versus least validity.
    3. Attach your filled-in PDF.

Unit 8: Assignment #3 (due before 11:59 pm Central on TUE JUL 17):

  1. Although the construct of validity is a general research methods term, when the construct of validity is applied to tests, such as IQ tests or personality inventories, we use the specific term “test validity.”
  2. To learn what the specific term “test validity” means and to be introduced to the major ways that test validity can be assessed, read a section from Heffner’s (no date) “Chapter 7.3 Test Validity.”
  3. Now, think back to one of the exams you took last semester in one of your courses. In fact, think about the most memorable exam you took (either because it was memorably positive or memorably negative). Refer to this exam as “Text X.”
  4. Go to the Unit 8: Assignment #3 Discussion Board and make a new post of at least 200 words in which you discuss Test X’s
    1. construct validity (did the exam measure what it was supposed to measure?);
    2. concurrent validity (was the exam like other comparable exams, for example, the exams your friends took in a similar course?);
    3. content validity (were the items on the test representative of the content you were expected to learn?); and
    4. predictive validity (how well did the exam predict your ability to use the information in the future?).

Unit 8: Assignment #4 (due before 11:59 pm Central on TUE JUL 17):

  1. Because we’re a bit more than halfway through the course, it’s a good time to revisit things you’ve learned so far in the course (which is why traditional lecture-based courses usually have midterm exams around the middle of the term).
  2. Listed below are Assignments you previously completed (or were supposed to complete). Choose one Assignment from the list that you enjoyed doing before and re-do that Assignment, according to the Assignment’s original instructions AND the additional requirements specified below.
    1. Unit 1: Assignment #3 (choose three situations that are different from the three situations you chose before and are different from the three situations that other students chose before)
    2. Unit 1: Assignment #5 (teach the Cognitive Biases to three people who are different from the three people you taught before)
    3. Unit 2: Assignment #5 (make a different teaching document than you made before, aimed at a different audience than you aimed at before and using a different medium than you used before)
    4. Unit 3: Assignment #3 (write your two essays on two quotes that are different from the two quotes you wrote essays on before; however, as before, one of your two essays must be based on a “Reasons/Arguments Prompt” quote and the other must be based on an “Examples Prompt” quote)
    5. Unit 3: Assignment #5 (write your two essays in response to two tasks that are different from the two tasks you completed before)
    6. Unit 4: Assignment #2 (find and analyze three different news stories than you found before)
    7. Unit 4: Assignment #3 (teach correlation is not causation to three people who are different from the three people you taught before)
    8. Unit 5: Assignment #1 (choose three topics that are different from the topics you chose; therefore, you’ll need to find three articles that are different from the articles you found before)
    9. Unit 5: Assignment #2 (find and analyze ten popular press articles that are different from the ten popular press articles you found before)
    10. Unit 7: Assignment #2 (make a different teaching document than you made before, aimed at a different audience than you aimed at before and using a different medium than you used before)
    11. Unit 7: Assignment #3 (choose a different set of three of your 9 articles than you analyzed before)
  3. Go to the Unit 8: Assignment #4 Discussion Board and make a new post containing everything that was previously required for the Assignment you’ve now re-done.
    1. Also, explain in just a sentence or two why you chose this Assignment to do again.
    2. Be sure to state somewhere in your post the Unit number and Assignment number (e.g., Unit 1: Assignment #3) of the Assignment you chose to do again.

Unit 8: Assignment #5 (due before 11:59 pm Central on WED JUL 18):

  1. Successful learning also requires revisiting previous challenges and overcoming those challenges.
  2. From the Assignments listed above, in Unit 8: Assignment #4, select one Assignment that you previously completed (or were supposed to complete) but you struggled (a bit or a lot) when completing it.
    1. Re-do that Assignment, according to the Assignment’s original instructions AND the additional requirements specified in the list in Unit 8: Assignment #4.
    2. One further constraint is that the Assignment you choose to re-do for Unit 8: Assignment #5 needs to be different from the Assignment you chose to re-do for Unit 8: Assignment #4.
  3. Go to the Unit 8: Assignment #5 Discussion Board and make a new post containing everything that was previously required for the assignment you’ve now re-done.
    1. Also, explain in just a sentence or two why you chose this Assignment to do again.
    2. Be sure to state somewhere in your post the Unit number and Assignment number (e.g., Unit 1: Assignment #3) of the Assignment you chose to do again.

Unit 8: Assignment #6 (due before 11:59 pm Central on WED JUL 18):

  1. At this point in the course we will construct new Chat Groups. Please follow closely the instructions below.
  2. If your last name is LAST alphabetically in your “old” Chat Group, these are your responsibilities for forming a new Chat Group:
    1. You’re responsible for recruiting new persons to your new Chat Group. You can contact students via their wisc.edu email (which is listed with their “old” Chat Group).
    2. You may recruit any student to your new Chat Group from your Section (but only from your Section) except
      • you may NOT recruit a student who was a member of your “old” Chat Group (because the goal during the second half of the course is to Chat with some different students than you worked with during the first half of the course); and
      • you may NOT recruit a student whose last name is also last alphabetically in their “old” Chat Group (because those students whose last names are last alphabetically in their old Chat Groups will also be the students responsible for recruiting students to a new Chat Group).
      • If you can’t remember what Section you are in, you might want to return to Unit 1: Assignment #2 (and the part of the assignment that said “From the Course How To, learn how to find out which section you are in and find out which section you are in”).
    3. If your “old” Chat Group had three members (including you), then your new Chat Group should also have three members (including you); if your “old” Chat Group had only two members, then your new Chat Group will have only two members (including you).
    4. When recruiting new  members to your new Chat Group:
      • It doesn’t matter if the students you are recruiting were previously in a two-member or three-member “old” Chat Group.
      • It only matters how many members you (the person whose last name comes LAST alphabetically and therefore the person doing the recruiting) had in your “old” Chat Group.
      • Similarly, it doesn’t matter if the students you are recruiting previously worked together in their “old” Chat Group; it only matters that none of the students you are recruiting are students you worked with before in your “old” Chat Group.
    5. Appoint one member of your new Chat Group to set up the new chat room (remind this person that the instructions for setting up a new chat room are below in c.).
    6. Write down
      • the names of the members of your new Chat Group (including you);
      • who (of the new member/s of your new Chat Group) you appointed to create the new chat room; and
      • the name of your new Chat Group (which will be the name of your “old” Chat Group).
  3. If you are appointed to set up the new chat room for your new Chat Group, these are your responsibilities:
    1. Follow the instructions in the Course How To for “How to Set Up a Small Group Text Chat.”
    2. Title the Google document the name of your new Chat Group (which will be the name of the old Chat Group of the student who recruited you to the new Chat Group; you are responsible for finding out that name).
    3. Take a screenshot of the name of your new Chat Group’s Google document (if you don’t know how to take a screenshot, this website will help you).
  4. If you are in a THREE-student NEW Chat Group AND you were NOT appointed to set up the new chat room for your new Chat Group, write down:
    1. the names of the members of your new Chat Group (including you);
    2. the name of your new Chat Group (which will be the name of the “old” Chat Group of the student who recruited you to the new Chat Group); and
    3. the name of the student who recruited you to the new Chat Group.
  5. EVERYONE: Continuing our theme of re-visiting at the middle of the term what you’ve learned so far in the course:
    1. Ensure that your Course Journal is up-to-date and contains entries for Units 1 through 8, as well as your Goals Setting Entry.
    2. Save your Course Journal as a PDF, titled LastName_CourseJournal_1-8.pdf.
    3. If your Course Journal is hand-written (or hand-drawn), you will need to scan all the pages into one PDF.
    4. Go to the Unit 8: Assignment #6 Discussion Board and make a new post to which you attach your Course Journal PDF; in addition, in the post itself, do one of the following:
      • If your last name came LAST alphabetically in your “old” Chat Group, post the names of the members of your new Chat Group (including you); who you appointed to create the new chat room; and the name of your new Chat Group.
      • If you are appointed to set up the new chat room for your new Chat Group, embed in your post the screenshot you took of the name of your new Chat Group’s Google document. Remember to re-size the screenshot appropriately (as you learned in the Course How To).
      • If you are in a THREE-student NEW Chat Group AND you were NOT appointed to set up the new chat room, post the names of the members of your new Chat Group (including you); the name of your new Chat Group; and the name of the student who recruited you to the new Chat Group.
    5. Lastly, your new Chat Group might want to arrange the time for the Group Chat you will need to hold during the next Unit (Unit 9: Assignment #6).

Congratulations, you have finished Unit 8! Onward to Unit 9!