Welcome!

We're very excited to welcome you to this advanced NLP seminar! We will cover ethical and social issues in NLP, especially focusing on large language models/foundation models. We'll cover research ethics, IRB, data and participatory research, harms and their mitigation, what models we should build and who should decide, and how to apply NLP to social questions about the justice system, framing and dehumanization, disinformation and toxicity, and for counter speech.

This is an advanced seminar with a lot of reading! We'll meet in breakout groups, and then bring the discussion back to the whole group! Attendence is strictly required, no hybrid option.

Logistics

Schedule

Each week of this class focuses either on ways to avoid ethical or social problems in doing NLP research (we call these Red Weeks “(NLP Should) Do No Harm”) or on ways to apply NLP to help solve social or ethical problems (we call these Blue weeks “(NLP Should) Do Good”). We've tried to give you extra papers beyond the 2-3 we read for each session, the extra papers are designed to be useful for helping come up with project ideas or literature surveys.

Week Date Description Course Materials Deadlines
1 April 6
Thursday
Part I: 3:00-4:00 Introduction to the class and each other [slides pptx, pdf]

Part II: 4:10-5:20 Where does the data come from? The Belmont Report, Participants, Labelers, and Data in NLP [slides pptx, pdf]
Required Readings: Plus either one of the following two papers:


Further Readings for Projects and Background

I'll be lecturing on these first two papers:

Emily M. Bender and Batya Friedman. 2018. Data statements for NLP: Toward mitigating system bias and enabling better science. TACL 6, 587–604.

Casey Fiesler and Nicholas Proferes. 2018. “Participant” Perceptions of Twitter Research Ethics. Social Media + Society, 4(1). 22


Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford. 2020.  Datasheets for Datasets.  Arxiv.

Vitak, J., Shilton, K., Beyond the Belmont principles: Ethical challenges, practices, and beliefs in the online data research community. In Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing (pp. 941-953).

Williams, M. L., Burnap, P., Towards an Ethical Framework for Publishing Twitter Data in Social Research: Taking into Account Users’ Views, Online Context and Algorithmic Estimation. Sociology, 51(6), 1149–1168.

Shuster, Evelyne. 1997. Fifty years later: the significance of the Nuremberg Code." New England Journal of Medicine 337, 20: 1436-1440.

The Common Rule:  The Federal Policy for the Protection of Human Subjects.  45 CFR part 46,

Kobi Leins and Jey Han Lau and Timothy Baldwin. 2020. Give Me Convenience and Give Her Death: Who Should Decide What Uses of NLP are Appropriate, and on What Basis?. ACL 2020

Rickford, John Russell. "Unequal partnership: Sociolinguistics and the African American speech community." Language in Society 26, no. 2 (1997): 161-197.

Read the Belmont plus one of the other papers before class. No need to write paragraphs for today.
2 April 13
Thursday
Part I: 3:00-4:00 The role of the local community, participatory research, and decolonization







Part II: 4:10-5:20 Harms of Language Models: Surveys and Background
Part I: Read any two of these three papers: Part II: Read these two

Langdon Winner. 1980.  “Do Artifacts have Politics?”, Daedalus,109 (1): 121-136

Leon Derczynski, Hannah Rose Kirk, Vidhisha Balachandran, Sachin Kumar, Yulia Tsvetkov, M. R. Leiser, Saif Mohammad. 2023. Assessing Language Model Deployment with Risk Cards.


Further Readings for Projects and Background:

Steven Bird. 2022. Local Languages, Third Spaces, and other High-Resource Scenarios. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7817–7829, Dublin, Ireland. ACL.

Shakir Mohamed, Marie-Therese Png, William Isaac, 2020. Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence. Philosophy and Technology 33: 659–684 .

Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, Mofetoluwa Adeyemi. 2022. Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets. Transactions of the Association for Computational Linguistics 2022; 10 50–72

Lane Schwartz. 2022. Primum Non Nocere: Before working with Indigenous data, the ACL must confront ongoing colonialism. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 724–731, Dublin, Ireland. Association for Computational Linguistics. https://aclanthology.org/2022.acl-short.82.pdf

Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2022. Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2340–2354, Dublin, Ireland. Association for Computational Linguistics. https://aclanthology.org/2022.findings-acl.184/

Surangika Ranathunga and Nisansa de Silva. 2022. Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP World. AACL/IJCNLP 2022, 823–848.

Heather Lent, Kelechi Ogueji, Miryam de Lhoneux, Orevaoghene Ahia, and Anders Søgaard. 2022. What a Creole Wants, What a Creole Needs. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6439–6449, Marseille, France. European Language Resources Association.

David Jurgens, Yulia Tsvetkov, and Dan Jurafsky. 2017. Incorporating Dialectal Variability for Socially Equitable Language Identification. ACL 2017.

Vithya Yogarajan, Gillian Dobbie, Henry Gouk. 20023. Effectiveness of Debiasing Techniques: An Indigenous Qualitative Analysis. ICLR TinyPaper 2023

Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John Rickford, Dan Jurafsky, and Sharad Goel. 2020. Racial Disparities in Automated Speech Recognition. Proceedings of the National Academy of Sciences 117 (14) 7684-7689.


Bender, Emily M., Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?🦜. FAccT '21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency March 2021 Pages 610–623

Weidinger, Laura, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2022. Taxonomy of risks posed by language models. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 214-229. 2022.

Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2021. Ethical and social risks of harm from Language Models. arXiv:2112.04359 [cs] (Dec. 2021).

Shelby, Renee, Shalaleh Rismani, Kathryn Henne, AJung Moon, Negar Rostamzadeh, Paul Nicholas, N'mah Yilla-Akbari, Jess Gallegos, Andrew Smart, Emilio Garcia, and Gurleen Virk. 2023. Identifying Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction. arXiv preprint arXiv:2210.05791 (2022).

A. Stevie Bergman, Gavin Abercrombie, Shannon Spruit, Dirk Hovy, Emily Dinan, Y-Lan Boureau, and Verena Rieser. 2022. Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 39–52, Edinburgh, UK. Association for Computational Linguistics.

Ganguli, Deep, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, Jack Clark. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858 (2022).

Read and post paragraphs by 5pm Wednesday April 12.
3 April 20
Thursday
Part I: 3:00-4:00 Harms of Language Models: Bias and Stereotype









Part II: 4:10-5:20 Harms of LLMs: Influence
Part I: Read two of these three papers:
  • Abubakar Abid, Maheen Farooqi, James Zou. 2021. Large language models associate Muslims with violence. Nature Machine Intelligence 3, 461-463 (2021).
  • Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, Diyi Yang. 2023. On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning . ArXiv.
  • Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamile Lukosiute, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, Dawn Drain, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jackson Kernion, Jamie Kerr, Jared Mueller, Joshua Landau, Kamal Ndousse, Karina Nguyen, Liane Lovitt, Michael Sellitto, Nelson Elhage, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Sandipan Kundu, Saurav Kadavath, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Christopher Olah, Jack Clark, Samuel R. Bowman, Jared Kaplan. 2023. The Capacity for Moral Self-Correction in Large Language Models. preprint. https://arxiv.org/abs/2302.07459
Part II: Read these 2 papers: Further Readings for Projects and Background:

  • Yotam Wolf, Noam Wies, Yoav Levine, Amnon Shashua. 2023. Fundamental Limitations of Alignment in Large Language Models.. ArXiv preprint.
  • Yang Cao, Anna Sotnikova, Hal Daumé III, Rachel Rudinger, and Linda Zou. 2022. Theory-Grounded Measurement of U.S. Social Stereotypes in English Language Models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1276–1295, Seattle, United States. Association for Computational Linguistics.
  • Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, James Zou. 2023. GPT detectors are biased against non-native English writers. Preprint at https://arxiv.org/abs/2304.02819
  • Julian Coda-Forno, Kristin Witte, Akshay K. Jagadish, Marcel Binz, Zeynep Akata, Eric Schulz. 2023. Inducing anxiety in large language models increases exploration and bias. ArXiv preprint.
  • Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2086–2105, Dublin, Ireland. Association for Computational Linguistics.
  • Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender Bias in Coreference Resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–14, New Orleans, Louisiana. Association for Computational Linguistics.
  • Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online. Association for Computational Linguistics.
  • Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5356–5371.
  • Debora Nozza, Federico Bianchi, and Dirk Hovy. 2021. HONEST: Measuring hurtful sentence completion in language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2398–2406, Online. Association for Computational Linguistics.
  • Dirk Hovy and Diyi Yang. 2021. The importance of modeling social factors of language: Theory and practice. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 588–602, Online. Association for Computational Linguistics
  • Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1004–1015, Online. Association for Computational Linguistics.
  • Wei Guo and Aylin Caliskan. 2021. Detecting emergent intersectional biases: Contextualized word embeddings contain a distribution of human-like biases. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’21, page 122–133, New York, NY, USA. Association for Computing Machinery.
  • Anna Sotnikova, Yang Trista Cao, Hal Daumé III, and Rachel Rudinger. 2021. Analyzing stereotypes in generative text inference tasks. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 4052–4065, Online. Association for Computational Linguistics.
  • Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967.
  • Nicholas Meade, Elinor Poole-Dayan, and Siva Reddy. 2022. An empirical survey of the effectiveness of debiasing techniques for pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1878–1898, Dublin, Ireland. Association for Computational Linguistics.
  • Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social Bias Frames: Reasoning about Social and Power Implications of Language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477–5490, Online. Association for Computational Linguistics.
  • Timo Schick, Sahana Udupa, and Hinrich Schütze. 2021. Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP. Transactions of the Association for Computational Linguistics, 9:1408–1424.
  • Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield- Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kaplan. 2022. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862
  • Josh A. Goldstein, Girish Sastry, Micah Musser, Renee DiResta, Matthew Gentzel, and Katerina Sedova. 2023. Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations. https://arxiv.org/abs/2301.04246
  • Bai, H., Voelkel, J. G., Eichstaedt, J. C., and Willer, R. (2023). Artificial Intelligence Can Persuade Humans on Political Issues. Preprint
  • Amalie Pauli, Leon Derczynski, and Ira Assent. 2022. Modelling Persuasion through Misuse of Rhetorical Appeals. In Proceedings of the Second Workshop on NLP for Positive Impact (NLP4PI), pages 89–100, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  • Luceri, Luca, Ashok Deb, Adam Badawy, and Emilio Ferrara. "Red Bots Do It Better: Comparative Analysis of Social Bot Partisan Behavior." WWW '19: Companion Proceedings of The 2019 World Wide Web Conference May 2019 Pages 1007–1012https://doi.org/10.1145/3308560.3316735
  • Adam Badawy, Kristina Lerman, and Emilio Ferrara. Who falls for online political manipulation? The Web Conference 2019 - Companion of the World Wide Web Conference, WWW 2019, pages 162–168, 2019.
  • Joseph Menn. April 16, 2023. Russians boasted that just 1% of fake social profiles are caught, leak shows.Washington Post.
  • Milano, Silvia, Mariarosaria Taddeo, and Luciano Floridi. 2020. Recommender systems and their ethical challenges. AI and Society 35 (2020): 957-967.
  • Benkler, Yochai, Robert Faris, and Hal Roberts. Network propaganda: Manipulation, disinformation, and radicalization in American politics. Oxford University Press, 2018.
  • Edward Bernays. 1922. Propaganda.
  • Read and post paragraphs by 5pm Wednesday April 18
    4 April 27
    Thursday
    Part I: 3:00-4:00 Harms of Language Models: Intellectual Property

















    Part II: 4:10-5:20 Harms of Language Models: Privacy and Libel

    Part I: Read these two papers:

    More papers for background:

    Part II: Read these 3 very short pieces (or in one case, a pair of pieces) and write 1 paragraph on Volokh/Kerr and 1 paragraph on the Italy case; don't write comments on the first reading (Section 5.4 "Legality"):

    More papers for background:
    Read and post paragraphs by 5pm Wednesday April 26.
    5 May 4
    Thursday
    Dan Sick: Class Cancelled
    6 May 11
    Thursday
    Part I: 3:00-4:00: Regulation (of Large Language Models, and AI in general)










    Part II: 4:10-5:20: The Role of the Technical Community: Should we build it? And other actions we can take.
    Part I: Do these readings and then instead of separate paragraphs on the readings, write the reflection as described in the Ed announcement More papers for background:

    Part II: Do these readings and then instead of separate paragraphs on the readings, write the reflection as described in the Ed announcement

    More papers for background:

    Read and post paragraphs by 5pm Wednesday May 10
    7 May 18
    Thursday
    Part I: 3:00-4:00: AI for the people

    Part II: 4:10-5:20: Dreaming up AI
    Part I: AI for the people

    Part II: Dreaming up AI

    • Read pages 68-70 and 76-78 of the People's Guide to AI (just do the reading, ignore the writing exercises))
    • Do at least one of these two lovely explorations:
      1. Have you and someone in your life go through this speculative sci-fi activity. See the instructions on Ed.
      2. Code something that brings you joy. It doesn't need to be useful; it can be beautiful; it can be anything. You don't need to share it with your group-mates afterward unless you want to. The only rules are: 1. it cannot be something you were going to do anyway or satisfies some requirement outside this class and 2. spend at least an hour on it; feel welcome to get carried away and spend more.
    Read and post paragraphs by 5pm Wednesday May 17. Lit Review due Mon May 15 5:00pm
    8 May 25
    Thursday
    Part I: 3:00-4:00: Using NLP for Social Change: Survey and Applications to Policing

















    Part II: 4:10-5:20: NLP for Social Good: Applications in Counter Speech
    Part I: Read these two papers: More papers for background:
    • Camp, Nicholas P., et al. "The thin blue waveform: Racial disparities in officer prosody undermine institutional trust in the police." Journal of personality and social psychology 121.6 (2021): 1157. https://www.apa.org/pubs/journals/releases/psp-pspa0000270.pdf
    • Eugenia H. Rho, Maggie Harrington, Yuyang Zhong, Reid Pryzant, Nicholas P. Camp, Dan Jurafsky, Jennifer L. Eberhardt. 2023. Escalated police stops of Black men are linguistically and psychologically distinct in their earliest moments. In Press, PNAS
    • Epp, Charles R., Steven Maynard‐Moody, and Donald Haider Markel. "Beyond profiling: The institutional sources of racial disparities in policing." Public Administration Review 77.2 (2017): 168-178. https://onlinelibrary.wiley.com/doi/full/10.1111/puar.12702
    • Elliott Ash, Daniel L. Chen, Arianna Ornaghi. Gender Attitudes in the Judiciary: Evidence from U.S. Circuit Courts. American Economic Journal: Applied Economics, forthcoming.
    • Erica Cai, Ankita Gupta, Katherine Keith, Brendan O'Connor, Douglas R. Rice. 2023. "Let Me Just Interrupt You": Estimating Gender Effects in Supreme Court Oral Arguments. SocArXiv

    Part II: Read this paper:

    More papers for background:
    • Dallas Card, Serina Chang, Chris Becker, Julia Mendelsohn, Rob Voigt, Leah Boustan, Ran Abramitzky, and Dan Jurafsky. 2022. Computational analysis of 140 years of US political speeches reveals more positive but increasingly polarized framing of immigration. Proceedings of the National Academy of Sciences 119 (31) e2120510119. Read pages 1-7, skip Materials and Methods except read the short "Measuring Dehumanization" section.
    • Sap, Maarten, Marcella Cindy Prasettio, Ari Holtzman, Hannah Rashkin, and Yejin Choi. 2017. "Connotation frames of power and agency in modern films."EMNLP 2017.
    • Charlesworth, Tessa, Aylin Caliskan, and Mahzarin R. Banaji. 2022. "Historical representations of social groups across 200 years of word embeddings from Google Books." Proceedings of the National Academy of Sciences 119, no. 28 (2022): e2121798119.
    • Nikhil Garg, Londa Schiebinger, Dan Jurafsky, James Zou. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences 2018.
    • Anjalie Field, Doron Kliger, Shuly Wintner, Jennifer Pan, Dan Jurafsky, and Yulia Tsvetkov. 2018. Framing and Agenda-setting in Russian News: a Computational Analysis of Intricate Political Strategies. EMNLP 2018
    • Rishi Bommasani and Percy Liang. 2023. Trustworthy Social Bias Measurement. Preprint
    • He, Bing, Mustaque Ahamad, and Srijan Kumar. 2023. "Reinforcement Learning-based Counter-Misinformation Response Generation: A Case Study of COVID-19 Vaccine Misinformation. WWW.
    Read and post paragraphs by 5pm Wednesday May 24. Project Proposal due Friday May 26 5:00pm
    9 May 29
    Thursday in classroom
    Individual meetings with Dan, Ria, Peter on projects.
    10 June 8
    Thursday
    Special Class Presentation Day (regular classes end June 7). Other days this week: more individual meetings with Dan, Ria, Peter on projects. Final Project Report due Mon June 12, 5:00pm