Master of Science in Marketing<\/a> at Drexel University\u2019s LeBow College of Business mirrors this industry shift. It is designed to prepare professionals for leadership at the cutting-edge intersection of marketing, data and technology.<\/p>\n\n\n\nThrough STEM-designated concentrations in digital marketing technology and marketing technology and analytics<\/strong>, Drexel LeBow\u2019s MS in Marketing provides critical skills with a foundation in data analysis, the scientific method and behavioral sciences. For example, students in the program learn how to:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Use data and scientific testing to optimize conversion rates.<\/li>\n\n\n\n<li>Apply behavioral science to anticipate interactions with marketing automation technology.<\/li>\n\n\n\n<li>Draw upon psychology, sociology and epistemology in content marketing to understand audience reactions.<\/li>\n<\/ul>\n\n\n\nKey courses in Drexel LeBow\u2019s MS in Marketing reinforce this industry emphasis on STEM, supporting students over a one- to two-year course of study as they engage in projects that call for data-led analysis, including:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Creating marketing plans that reflect known consumer psychology principles and behaviors.<\/li>\n\n\n\n<li>Projecting ROI and customer lifetime value to make data-informed recommendations on strategy.<\/li>\n\n\n\n<li>Studying consumer behavior through case studies, which may include experiments in LeBow\u2019s Behavioral Lab, equipped with eye-tracking devices, neurological monitors and other tools for gathering biometric data.<\/li>\n\n\n\n<li>Using tools like Google Analytics and Tableau, along with developing skills and best practices to enhance user experience.<\/li>\n<\/ul>\n\n\n\nIn today\u2019s digital-first, data-informed landscape, marketers must evolve to stay ahead. Drexel LeBow\u2019s MS in Marketing provides students with the tools to create, innovate and thrive in their careers.<\/p>\n\n\n\n<a class=\"wp-block-ama-download button button-solid button-red\" href=https://www.ama.org/"https:////www.lebow.drexel.edu//land//masters//marketing?utm_source=ll_ama&utm_medium=native&utm_campaign=ll_drx_leb_ms_mkt_fy25&utm_content=ama_content&utm_term=dom_pros_native\%22 download>LEARN MORE<\/a>\n","post_title":"Where Data Dominates, Marketing and STEM Work Hand in Hand","post_excerpt":"","post_status":"publish","comment_status":"closed","ping_status":"closed","post_password":"","post_name":"where-data-dominates-marketing-and-stem-work-hand-in-hand","to_ping":"","pinged":"","post_modified":"2025-03-19 15:25:30","post_modified_gmt":"2025-03-19 20:25:30","post_content_filtered":"","post_parent":0,"guid":"https:\/\/www.ama.org\/?p=187403","menu_order":0,"post_type":"post","post_mime_type":"","comment_count":"0","filter":"raw"},{"ID":183062,"post_author":"203881","post_date":"2025-02-05 13:53:37","post_date_gmt":"2025-02-05 19:53:37","post_content":"\n\n\nThe <a href=https://www.ama.org/"https:////gdpr-info.eu///" target=\"_blank\" rel=\"noreferrer noopener\">General Data Protection Regulation (GDPR)<\/a> introduced by the European Union in 2018 marked a pivotal moment in privacy regulation, especially for online advertising practices. A <a href=https://www.ama.org/"https:////doi.org//10.1177//00222437231171848/" target=\"_blank\" rel=\"noreferrer noopener\">2024 Journal of Marketing Research<\/em> study<\/a> by Pengyuan Wang, Li Jiang, and Jian Yang investigates the early effects of GDPR compliance on display advertising, focusing on a large United States\u2013based publisher with global traffic using a pay-per-click (PPC) model. By analyzing a proprietary dataset containing over 3.7 billion ad impressions across 6,000 ad creatives and multiple industries, the study provides insights into GDPR's implications for ad performance, revenue, and the potential of contextual targeting as a mitigating strategy.<\/p>\n\n\n\nGDPR requires explicit user consent for personal data usage, fundamentally altering the landscape of behavioral targeting. Pre-implementation forecasts anticipated significant revenue declines, with some predicting annual ad revenue losses of up to 17%.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-gdpr-reduced-ad-performance\">GDPR Reduced Ad Performance <\/h2>\n\n\n\nWang et al. applied a difference-in-differences (DID) methodology to assess GDPR's impact, comparing data from EU users (treatment group) with non-EU users (control group) during five-week periods before and after April 18, 2018, when GDPR compliance was adopted. Key metrics analyzed as dependent variables included click-through rates (CTR), conversion rates, bid prices, and revenue per click.<\/p>\n\n\n\nThe results revealed moderate but significant declines in ad performance and revenue following GDPR compliance. Revenue per click dropped by 5.7%, primarily due to reduced bid prices and fewer active advertisers. The CTR and conversion rates decreased by 2.1% and 5.4%, respectively, signalling diminished user engagement and conversion efficiency. These findings highlight the challenges posed by restricted access to personal data in ad targeting.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-content-and-industry-factors\">Content and Industry Factors<\/h2>\n\n\n\nThe study identifies content-based targeting as a mitigating factor. Ads contextually aligned with webpage themes, such as sports ads on sports-related pages, were less impacted by the GDPR. Contextual targeting offset approximately 44% of the conversion rate decline and 42% of the revenue-per-click loss caused by the absence of personal data. This underscores the strategic value of content-based targeting as a privacy-compliant alternative, offering a degree of resilience for publishers navigating GDPR\u2019s constraints.<\/p>\n\n\n\nThe study also highlights varying impacts across industries. Sectors such as travel and financial services experienced greater performance decline compared to retail and consumer packaged goods (CPG). This disparity reflects differences in reliance on granular audience segmentation, with industries requiring precise targeting (such as travel and finance) being more adversely affected by GDPR's restrictions.<\/p>\n\n\n\nDespite these declines, the advertiser return on investment (ROI) under the PPC model remained stable. This stability suggests that publishers rather than advertisers endured most of GDPR's economic effects. Reduced bid prices primarily impacted publisher revenues, while advertisers continued to pay proportional costs per click.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote\">\nReduced bid prices primarily impacted publisher revenues, while advertisers continued to pay proportional costs per click.<\/p>\n<\/blockquote>\n\n\n\nThrough robust analysis and a large real-life dataset, Wang et al. contribute valuable insights at the crossroads of privacy regulations, targeting strategies, and economic outcomes in display advertising. Their findings quantify GDPR\u2019s initial impact and highlight the resilience and adaptability of advertising practices in response to stringent privacy norms. As policymakers and industry professionals navigate the implications of GDPR and similar regulations, this study offers a roadmap for managing challenges and leveraging opportunities in the evolving digital advertising landscape. <\/p>\n\n\n\nFor more insights, check out this interview with lead author Pengyuan Wang:<\/strong><\/p>\n\n\n\nQ: Your paper highlights how a limitation in access to personal data only caused a limited negative impact on ad performance, bid prices, and ad revenue. Do you think personal data in online advertising contexts is overrated by the industry?<\/em><\/strong><\/p>\n\n\n\nA:<\/strong> I don\u2019t think that personal data is overrated by the industry. The entire internet ecosystem and the free content model we all benefit from are, to some extent, built based on online advertising. The efficiency and revenue growth of online advertising have been significantly enhanced by the use of personal data. Personal data acted as a catalyst, not only for online advertising but also for the development of the IT industry.<\/p>\n\n\n\nWe have been using personal data for years, and now we are moving toward a more privacy-conscious era to\u00a0ensure that our personal data is used responsibly and aligns with each user\u2019s preference. This is good and necessary, but it does not eliminate the critical role of personal data in online advertising.<\/p>\n\n\n\nQ: The results of the effect of personal data found in the literature vary significantly in magnitude. Do you find this surprising and why do you think that is?<\/em><\/strong><\/p>\n\n\n\nA:<\/strong> It is not surprising to me. Companies use personal data in different ways. The impact of personal data is highly context-dependent and is affected by factors such as the industry, nature of the advertising campaigns, and the sophistication of the algorithms used to analyze and apply the data.<\/p>\n\n\n\nQ: Your work distinguishes itself from the previous research by using a natural experiment instead of self-reported purchase intentions. Why in your opinion is there such a discrepancy between the self-reported data and actual measures?<\/em><\/strong><\/p>\n\n\n\nA:<\/strong> Both self-reported data and actual measures provide valuable information, and I don\u2019t think one is inherently superior to the other. As for the discrepancy between studies based on self-reported data and actual measures, again, all studies have different contexts, which can significantly affect the outcomes. In addition, there is a fundamental distinction between what people say and do. Privacy, by itself, is important. However, when people need to choose between free content and privacy, there is a trade-off to make. Thus, personally I think self-reported data might tend to overestimate the importance of privacy protection.<\/p>\n\n\n\nQ: The study spans five weeks before and after GDPR compliance. Are there indications of how GDPR's impact might evolve over a longer period? Do you anticipate that publishers will adopt new strategies as they adapt to the GDPR constraints?<\/em><\/strong><\/p>\n\n\n\nA:<\/strong> During the study period, I did not observe any clear indications of GDPR\u2019s impact evolving over time. I do expect publishers to adopt new strategies gradually, but it takes time for publishers to develop and experiment with these new strategies.<\/p>\n\n\n\nQ: The study notes a decrease in both bid prices and number of active advertisers. Could you discuss the potential implications of this trend for market competitiveness and pricing strategies in online advertising?<\/em><\/strong><\/p>\n\n\n\nA:<\/strong> During the study period, the observed decreases in bid prices and number of active advertisers suggest reduced market competitiveness. This finding suggests that advertisers may be less willing to invest in online advertising when tighter regulations on personal data are implemented. However, I do not think it will always be like this. Online advertising remains a powerful tool, and the industry is evolving too to develop ad strategies that rely less on personal data or to use people\u2019s personal data in ways that still protect privacy. For example, there are papers on how to use data anonymization techniques (such as k-anonymization) for online advertising, which allows for user targeting while protecting individual privacy. Many other approaches such as differential privacy and federated learning are also under development for online advertising. I am confident that online advertising will continue to thrive as the industry adapts to these challenges.<\/p>\n\n\n\nQ: Do you think CPC pricing models are more impacted by limited access to personal data compared to CPM? Could it be inferred that CPA also suffers\u00a0from a lack of behavioral-based data? In other words, does this impact conversion-based campaigns compared to branding and awareness campaigns? Could that explain why Google has many times delayed the plan for cookie deprecation, given their core business on search ads?<\/em><\/strong><\/p>\n\n\n\nA:<\/strong> The company in this study employed CPC pricing models, so I don\u2019t have data on the impact of GDPR on the CPM model. My conjecture is that stricter regulations would protect user privacy at the cost of ad efficiency. This will also affect CPM models, as it may reduce the likelihood of reaching high-intent buyers with the same number of impressions.<\/p>\n\n\n\nFor the CPA model, again, I don\u2019t have data from businesses operating under this model, but I can offer some conjectures. If we define the \u201cA\u201d (action) here as something very close to a purchase, I conjecture that the CPA metric itself might be stable, because CPA represents the value of an action, which should be stable. However, it might be more difficult to perform the action. So CPA might be stable, but the total number of users completing these actions could decrease, ultimately leading to lower revenue. Therefore, I think that companies with CPA models would also see some impact.<\/p>\n\n\n\nAs for Google\u2019s decision to delay its plans for cookie deprecation, I cannot comment on their internal business decisions since I don\u2019t have access to this information. That being said, cookies have historically played a significant role in collecting user information and enhancing Google\u2019s advertising efficiency. The delay might reflect the challenges of transitioning away from such a tool while maintaining ad effectiveness.<\/p>\n\n\n\nQ: We often discuss the different approaches taken by Europe and the United States in regulating technology and its impact on innovation and the economy. Do you think GDPR-like regulations stifle or foster innovation?<\/em><\/strong><\/p>\n\n\n\nA:<\/strong> I believe regulations such as GDPR need to be implemented at the right time. In the early stages of the internet and online advertising industries, when companies were still developing ways to leverage personal data to enhance ad efficiency, when they needed to quickly generate revenue and accumulate capital for re-investment, stricter regulations could have hampered growth. For example, if the GDPR was\u00a0implemented in 2008, it might have stifled innovation and slowed the development of the broader internet economy.<\/p>\n\n\n\nHowever, by 2018, when GDPR became effective, online advertising reached a certain level of maturity. The industry had accumulated sufficient capital and expertise, allowing companies to bear temporary impacts on revenue and invest in new technologies. This timing provided an opportunity for innovation in privacy-preserving ad strategies such as data anonymization or federated learning.<\/p>\n\n\n\nTherefore, restrictive regulations can hinder innovation if introduced prematurely. When they are applied at the right stage of industry maturity, they could drive the development of creative solutions and ethical practices.<\/p>\n\n\n\n\n\nSource:<\/strong> Pengyuan Wang, Li Jiang, and Jian Yang (2024), \"<a href=https://www.ama.org/"https:////doi.org//10.1177//00222437231171848/" target=\"_blank\" rel=\"noreferrer noopener\">The Early Impact of GDPR Compliance on Display Advertising: The Case of an Ad Publisher<\/a>,\" <a href=https://www.ama.org/"https:////www.ama.org//journal-of-marketing-research///" target=\"_blank\" rel=\"noreferrer noopener\">Journal of Marketing Research<\/a><\/em>, 61 (1), 70\u201391. <a href=https://www.ama.org/"https:////doi.org//10.1177//00222437231171848/" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/doi.org\/10.1177\/00222437231171848<\/a><\/p>\n\n\n\nGo to the <a href=https://www.ama.org/"https:////www.ama.org//journal-of-marketing-research///" target=\"_blank\" rel=\"noreferrer noopener\">Journal of Marketing Research<\/a><\/em><\/p>\n\n\n\n<div style=\"height:40px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n\n\n<div style=\"height:40px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n","post_title":"How GDPR Changed the Game for Display Advertising","post_excerpt":"A Journal of Marketing Research study finds that GDPR led to significant declines in ad performance, but content-based targeting may be the answer.","post_status":"publish","comment_status":"closed","ping_status":"closed","post_password":"","post_name":"how-gdpr-changed-the-game-for-display-advertising","to_ping":"","pinged":"","post_modified":"2025-02-05 16:12:41","post_modified_gmt":"2025-02-05 22:12:41","post_content_filtered":"","post_parent":0,"guid":"https:\/\/www.ama.org\/?p=183062","menu_order":0,"post_type":"post","post_mime_type":"","comment_count":"0","filter":"raw"},{"ID":181130,"post_author":"124160","post_date":"2025-01-14 05:00:00","post_date_gmt":"2025-01-14 11:00:00","post_content":"\nGenerative AI (GenAI), and large language models (LLMs) in particular, are transforming marketing. According to a <a href=https://www.ama.org/"https:////www.bcg.com//publications//2023//generative-ai-in-marketing/" target=\"_blank\" rel=\"noreferrer noopener\">2023 BCG study<\/a>, over 70% of chief marketing officers have embraced this technology, and experts predict GenAI to revolutionize marketing research\u2014a $84.3 billion dollar industry in 2023\u2014by automating and enhancing data collection, analysis, and insights generation. <\/p>\n\n\n\nIn a <a href=https://www.ama.org/"https:////doi.org//10.1177//00222429241276529/" target=\"_blank\" rel=\"noreferrer noopener\">new Journal of Marketing<\/em> study<\/a>, we find that LLMs offer significant efficiency and effectiveness gains in the marketing research process for both qualitative and quantitative research. We show that LLMs serve as excellent assistants for insights managers through different stages of the research process: study design, sample selection, data collection, and data analysis. <\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-the-ai-human-hybrid-approach-nbsp-nbsp\">The AI-Human Hybrid Approach <\/strong> <\/h2>\n\n\n\nConsider a business context in which a brand manager collaborates with a consumer insights manager to formulate the problem the research is trying to address and come up with a set of research questions. The two may collaboratively agree on a research design that, for example, begins with exploratory research (e.g., in-depth interviews) followed by descriptive research (e.g., a survey). These first two steps of the research process are largely led by humans. Although the brand and insight managers could consult an LLM to gather secondary research on the topic and explore use cases that could help inform the research questions or research design, they would still largely rely on their knowledge of the business context to formulate the research problem, questions, and design. <\/p>\n\n\n\nOur central premise is that a human\u2013LLM hybrid approach can lead to efficiency and effectiveness gains in the marketing research process. For this study, we partnered with a Fortune 500 food company and replicated two studies the company had conducted using an LLM. The first study was qualitative and centered around business questions for the Friendsgiving celebration. The second study focused on testing a new refrigerated dog food. For each study, we treated the original (human) studies as the \u201cground truth\u201d and benchmarked the LLM generated studies against them. This approach allowed us to objectively evaluate the quality of synthetic data and investigate the role LLMs could play in knowledge generation.<\/p>\n\n\n\nFor qualitative research, we find that LLMs are excellent assistants for data generation and analysis. <\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>On the data generation front, LLMs effectively create desirable sample characteristics, generate synthetic respondents that match those characteristics, and conduct and moderate in-depth interviews. Our results show that LLM-generated responses are superior in terms of depth and insightfulness. <\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>On the analysis front, LLMs perform well, matching human experts in identifying key ideas, grouping them into themes, and summarizing information. Although LLMs missed some themes that humans detected, they generated some that humans did not. Expert judges find that human\u2013LLM hybrids outperformed their human-only or LLM-only counterparts. The upshot here is that LLMs and humans bring unique, complementary insights that managers should leverage. <\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-a-handy-research-assistant-nbsp\">A Handy Research Assistant<\/strong> <\/h2>\n\n\n\nAn LLM can be an excellent starting point for creating the first draft of a survey and can generate survey introductions, screener questions, and demographic questions with relative ease. The LLM can focus on the laborious, repetitive, and uninteresting tasks while the human expert can use this time savings to think more creatively about answers to the business questions and the quality of the insights. <\/p>\n\n\n\nA significant advantage of LLMs as an assistant is their low cost. We believe that this single factor will contribute toward rapid adoption of LLMs for insight generation. The gains here are likely to be higher for hard-to-reach respondents (e.g., doctors, senior managers) because synthetic respondents do not get tired and can provide lengthy answers to many questions. In the B2B arena where the end users and buyers are not easy to reach, LLMs could be quite helpful in supplementing the information gathered from human respondents. As an intelligent engine, an LLM could be a revolutionary generator of prior information for a wide variety of business questions at a low cost. <\/p>\n\n\n\nIt is important to note that LLMs can be wrong, biased, or hallucinate when not trained on the relevant data. Therefore, a human supervisor is a necessary part of the marketing research knowledge production process. For example, the human can make decisions about when not to ask an LLM for help. This could occur when the information sought is new not only to the company but also to the world. Other examples include marketing research in cultural contexts to understand local customs and traditions, topics with ethical considerations such as targeting vulnerable populations, and obtaining insights from data containing personal information, where LLMs may lack the necessary safeguards for data security and privacy. <\/p>\n\n\n\n\n\nSource: <\/strong>Neeraj Arora, Ishita Chakraborty, and Yohei Nishimura, \u201c<a href=https://www.ama.org/"https:////doi.org//10.1177//00222429241276529/" target=\"_blank\" rel=\"noreferrer noopener\">AI-Human Hybrids for Marketing Research: Leveraging LLMs as Collaborators<\/a>,\u201d <a href=https://www.ama.org/"https:////www.ama.org//journal-of-marketing///" target=\"_blank\" rel=\"noreferrer noopener\">Journal of Marketing<\/a><\/em><\/em>.<\/p>\n\n\n\nGo to the <a href=https://www.ama.org/"https:////www.ama.org//journal-of-marketing///" target=\"_blank\" rel=\"noreferrer noopener\">Journal of Marketing<\/a><\/em><\/p>\n\n\n\n<div style=\"height:40px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n\n\n<div style=\"height:40px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n\n\n<\/p>\n","post_title":"The Ultimate Research Assistant: How Marketing Researchers Can Effectively Collaborate with LLMs","post_excerpt":"This Journal of Marketing study highlights the effectiveness of an AI\u2013human hybrid approach in marketing research. LLMs with human oversight are valuable collaborators across different stages of qualitative and quantitative research.","post_status":"publish","comment_status":"closed","ping_status":"closed","post_password":"","post_name":"the-ultimate-research-assistant-how-marketing-researchers-can-effectively-collaborate-with-llms","to_ping":"","pinged":"","post_modified":"2025-01-13 18:22:41","post_modified_gmt":"2025-01-14 00:22:41","post_content_filtered":"","post_parent":0,"guid":"https:\/\/www.ama.org\/?p=181130","menu_order":0,"post_type":"post","post_mime_type":"","comment_count":"0","filter":"raw"},{"ID":179838,"post_author":"201023","post_date":"2025-01-07 05:00:00","post_date_gmt":"2025-01-07 11:00:00","post_content":"\nConsider a landscaping company whose designs focus on native plants and water conservation. The company creates two advertisements: one focused on sustainability (ad A) and another on aesthetics (ad B). As platforms personalize the ads that different users receive, ads A and B will be delivered to groups with diverging mixes. Users interested in outdoor activities may see the sustainability ad, whereas users interested in home decor may see the aesthetics ad. Targeting ads to specific consumers is a major part of the value that platforms offer to advertisers because it aims to place the \"right\" ads in front of the \"right\" users.<\/p>\n\n\n\nIn a <a href=https://www.ama.org/"https:////doi.org//10.1177//00222429241275886/" target=\"_blank\" rel=\"noreferrer noopener\">new Journal of Marketing<\/em> study<\/a>, we find that online A-B testing in digital advertising may not be delivering the reliable insights marketers expect. Our research uncovers significant limitations in the experimentation tools provided by online advertising platforms, potentially creating misleading conclusions about ad performance.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-the-issue-with-divergent-delivery\">The Issue with Divergent Delivery<\/h2>\n\n\n\nWe highlight a phenomenon called \"divergent delivery,\" in which the targeting algorithms used by online advertising platforms like Meta and Google target different types of users with different ad content. The problem arises when the algorithm sends different ads to distinct mixes of users using A-B testing: an experiment designed to compare the effectiveness of the two ads. The \u201cwinning\u201d ad may have performed better simply because the algorithm showed it to users who were more prone to respond to the ad than the users who saw the other ad. The same ad could appear to perform better or worse depending on the mix of users who see it rather than on the creative content of the ad itself.<\/p>\n\n\n\nFor an advertiser, especially with a large audience to choose from and a limited budget, targeting provides plenty of value. So large companies like Google and Meta use algorithms that allocate ads to specific users. On these platforms, advertisers bid for the right to show ads to users in an audience. However, the winner of an auction for the right to place an ad on a particular user\u2019s screen is not based on monetary value of the bids alone but also the ad content and user\u2013ad relevance. The precise inputs and methods that determine the relevance of ads to users, how relevance influences auction results, and, thus, which users are targeted with each ad, are proprietary to particular platforms and are not observable to advertisers. It is not precisely known how the algorithms determine relevance for types of users and it may not even be able to be enumerated or reproduced by the platforms themselves.<\/p>\n\n\n\nOur findings have profound implications for marketers who rely on A-B testing of their online ads to inform their marketing strategies. Because of low cost and seemingly scientific appeal, marketers use these online ad tests to develop strategies even beyond just deciding what ad to include in the next campaign. So, when platforms do not explicitly state that these experiments are not truly randomized, it gives marketers a false sense of security about their data-driven decisions.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-a-fundamental-problem-with-online-advertising\">A Fundamental Problem with Online Advertising<\/h2>\n\n\n\nWe argue that this issue is not just a technical flaw in this tool but a fundamental characteristic of how the online advertising business operates. The platform's primary goal is to maximize ad performance, not to provide experimental results for marketers. Therefore, these platforms have little incentive to let advertisers untangle the effect of ad content from the effect of their proprietary targeting algorithms. Marketers are left in a difficult position in that they must either accept the confounded results from these tests or invest in more complex and costly methods to truly understand the impact of creative elements in their ads.<\/p>\n\n\n\nOur study makes its case using simulation, statistical analysis, and a demonstration of divergent delivery from an actual A-B test run in the field. We challenge the common belief that results from A-B tests that compare multiple ads provide the same ability to draw causal conclusions as do randomized experiments. Marketers should be aware that the differences in effects of ads A and B that are reported by these platforms may not fully capture the true impact of their ads. By recognizing these limitations, marketers can make more informed decisions and avoid the pitfalls of misinterpreting data from these tests.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-advice-for-advertisers\">Advice for Advertisers<\/h2>\n\n\n\nWe offer the following recommendations for those using A-B testing tools:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img src=https://www.ama.org/"https:////www.ama.org//wp-content//uploads//2025//01//AB-testing-figure.jpg?resize=1024,285\%22 alt=\"\" class=\"wp-image-179882\"\/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>If your goal is to predict which ad creatives will perform best in a targeted <\/em>environment\u2014under the same conditions on the same ad platform with the same campaign setting\u2014 our advice is to carry on using the available A-B testing tools. Experimenters with this goal may not mind\u2014and even may prefer\u2014that their A-B tests lack balance across ad creative treatments and lack representativeness of the subjects.<\/li>\n\n\n\n<li>If the goal is to learn how different ad creatives generate different responses more generally, the report of the test should include the disclaimer that the A-B comparisons were made on a subset of the audience, across different mixes of users optimized for each ad separately, where subjects were selected by the proprietary algorithm.<\/li>\n\n\n\n<li>If the marketing objective is to extrapolate comparisons between ad content for use outside of the current platform (e.g., marketing strategy development, or offline advertising where randomized experimentation and user tracking is more challenging), our advice is to not rely on these A-B tests for causal evidence about the effects of creative content across ads. The analytics team, for instance, should warn that results are confounded by how the algorithm determined which ad treatments were most relevant to different experimental subjects. These disclosures should also be made by academic researchers who use A-B test results for scientific inference.<\/li>\n<\/ul>\n\n\n\nTo summarize, an A-B test may appear to be an easy way to run field experiments to learn about the effects of ads, imagery, and messaging. But experimenters who run A-B tests in targeted online advertising environments should know what they are really getting. Our concern is not the mere usage of certain types of A-B tests. Rather, it is the presentation of results as if they came from balanced experiments and subsequent conclusions and managerial decisions based on those results.<\/p>\n\n\n\n\n\nSource: <\/strong>Michael Braun and Eric M. Schwartz, \u201c<a href=https://www.ama.org/"https:////doi.org//10.1177//00222429241275886/" target=\"_blank\" rel=\"noreferrer noopener\">Where A-B Testing Goes Wrong: How Divergent Delivery Affects What Online Experiments Cannot (and Can) Tell You About How Customers Respond to Advertising<\/a>,\u201d <a href=https://www.ama.org/"https:////www.ama.org//journal-of-marketing///" target=\"_blank\" rel=\"noreferrer noopener\">Journal of Marketing<\/a><\/em>.<\/p>\n\n\n\nGo to the <a href=https://www.ama.org/"https:////www.ama.org//journal-of-marketing///" target=\"_blank\" rel=\"noreferrer noopener\">Journal of Marketing<\/a><\/em><\/p>\n\n\n\n<div style=\"height:40px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n\n\n<div style=\"height:40px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n","post_title":"Can You Trust Your Ad Data? A New Study Exposes a Hidden Flaw in A-B Testing on Digital Ad Platforms","post_excerpt":"A Journal of Marketing study shows how the experimentation tools provided by online advertising platforms can lead to misleading conclusions about ad performance.","post_status":"publish","comment_status":"closed","ping_status":"closed","post_password":"","post_name":"can-you-trust-your-ad-data-a-new-study-exposes-a-hidden-flaw-in-a-b-testing-on-digital-ad-platforms","to_ping":"","pinged":"","post_modified":"2025-01-07 10:33:17","post_modified_gmt":"2025-01-07 16:33:17","post_content_filtered":"","post_parent":0,"guid":"https:\/\/www.ama.org\/?p=179838","menu_order":0,"post_type":"post","post_mime_type":"","comment_count":"0","filter":"raw"},{"ID":175745,"post_author":"146335","post_date":"2024-11-11 10:35:25","post_date_gmt":"2024-11-11 16:35:25","post_content":"\n<h4 class=\"wp-block-heading\" id=\"h-let-s-take-a-look-at-how-synthetic-data-is-different-how-it-s-being-validated-and-how-it-is-and-should-be-used-as-a-strategic-tool-designed-to-bridge-significant-gaps-in-the-traditional-research-process\">Let\u2019s take a look at how synthetic data is different, how it\u2019s being validated, and how it is and should be used as a strategic tool designed to bridge significant gaps in the traditional research process.<\/h4>\n\n\n\nUnlock the future of market research with Yabble\u2019s Guide to Synthetic Data in the Real World of Research! Discover how synthetic data is transforming insights by enhancing data quality, reducing biases, and accelerating access to actionable insights.<\/p>\n\n\n\nWhat\u2019s Inside? Learn about:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Superior Data Quality & Accuracy: Yabble\u2019s synthetic data is rooted in real-world information, ensuring accuracy without the biases of traditional survey panels<\/li>\n\n\n\n<li>Instant Insights with Virtual Audiences: Skip the extensive fieldwork! Synthetic data offers immediate, comprehensive insights, so you can dive straight into strategy.<\/li>\n\n\n\n<li>Addressing Traditional Challenges: Synthetic data fills gaps left by conventional methods, overcoming issues like low-quality responses and dataset limitations.<\/li>\n\n\n\n<li>Real-World Solutions, Ready to Use: Get started today with low-risk, cost-effective solutions that deliver results.<\/li>\n<\/ul>\n\n\n\nSynthetic data is more than a trend \u2013 it\u2019s the competitive edge marketers need. Yabble\u2019s advanced language models pull from a rich mix of data sources to ensure insights are not only accurate but minimized for bias, empowering brands to make well-informed, data-backed decisions confidently.<\/p>\n\n\n\n<a class=\"wp-block-ama-download button button-solid button-red\" href=https://www.ama.org/"https:////ama.tradepub.com//c//pubRD.mpl?secure=1&sr=pp&_t=pp:&qf=w_defa7434&ch=\%22 download>Download<\/a>\n","post_title":"Yabble Guide to Synthetic Data in the Real World of Research","post_excerpt":"","post_status":"publish","comment_status":"closed","ping_status":"closed","post_password":"","post_name":"yabble-guide-to-synthetic-data-in-the-real-world-of-research","to_ping":"","pinged":"","post_modified":"2024-11-11 14:41:48","post_modified_gmt":"2024-11-11 20:41:48","post_content_filtered":"","post_parent":0,"guid":"https:\/\/www.ama.org\/?p=175745","menu_order":0,"post_type":"post","post_mime_type":"","comment_count":"0","filter":"raw"},{"ID":175179,"post_author":"198507","post_date":"2024-11-20 09:58:32","post_date_gmt":"2024-11-20 15:58:32","post_content":"\nIn this article, we attempt to overview the methodological toolkit available to empirical researchers who are interested in making causal inference using quasi-experimental data. In particular, Figure 1 provides an overview of the type of data available to researchers (e.g., randomized treatment, rich or constrained availability of observables, small or large number of time periods or treatment units) and describes corresponding suitable approaches, along with some pros and cons involved in their tactical use.<\/p>\n\n\n\nIn marketing, randomized experimentation represents the gold standard for making causal inference using empirical data. In an ideal setting, we would randomly assign participants to different groups to receive varying types or levels of treatment. A rich history of research, including work published in the <a href=https://www.ama.org/"https:////www.ama.org//journal-of-marketing-research///" target=\"_blank\">Journal of Marketing Research<\/a><\/em>, has used randomized experimental designs for causal inference (see <a href=https://www.ama.org/"https:////doi.org//10.1177//00222437231180494/" target=\"_blank\">Ghose et al. [2024]<\/a> and <a href=https://www.ama.org/"https:////doi.org//10.1177//00222437221131562/" target=\"_blank\">Cao, Chintagunta, and Li [2023]<\/a> for recent examples). Nonetheless, there are many marketing-relevant settings where researchers do not have access to experimental data, or where running such experiments is too expensive or infeasible. Additionally, ethical considerations frequently preclude randomly assigning treatments, such as instances where it could lead to harm or deprive participants of necessary care, as in the case of life-saving treatments or medications. When randomization of treatment is not possible, one may have to rely on enhanced \u201cstatistical rigor\u201d to compensate for the deficiencies in \u201cdesign rigor.\u201d<\/p>\n\n\n\n\n\nIn the remainder of this article, we review the common challenges pertaining to making causal inference with quasi-experimental data and discuss recent advances in helping alleviate them. Specifically, we focus on discussing methods that emphasize matching units based on the outcome variable <\/em>(Y<\/strong>). Then, we provide an overview of methods that focus on matching on observable covariates <\/em>(X<\/strong>). Lastly, we conclude by reflecting on our recommendations and discussing future research in related areas.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-difference-in-differences\"><a><\/a>Difference-in-Differences<\/h2>\n\n\n\nIn (the many) situations where researchers do not have the luxury of assigning units into treatment and control groups, they can still understand causal effects by leveraging quasi-experimental methods. The difference-in-differences (DID) method is the most widely used quasi-experimental method. It can be used in data settings with treatment and control units and pre- and post-treatment time periods.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img src=https://www.ama.org/"https:////www.ama.org//wp-content//uploads//2024//11//Li-Fig-1.png/" alt=\"\" class=\"wp-image-176024\"\/><figcaption class=\"wp-element-caption\">Figure 1: Overview of Design Choices in Quasi-Experimental Settings<\/strong><\/figcaption><\/figure>\n\n\n\nHere, we begin by describing the simplest DID design where all treatment units are treated at the same time. We observe treatment and control units over time so yit<\/sub> is the outcome for unit i at time t. The DID model can be estimated using the following regression model:<\/p>\n\n\n\n<div class=\"wp-block-columns is-not-stacked-on-mobile\">\n<div class=\"wp-block-column\" style=\"flex-basis:66.66%\">\nyit<\/sub> = \u03b21<\/sub> + \u03b22<\/sub>Treati<\/sub> + \u03b23<\/sub>Postt<\/sub> + \u03b24<\/sub>Treati<\/sub>Postt<\/sub> + xi<\/sub>\u03b2\u02dc + \u03f5it<\/sub>.<\/p>\n<\/div>\n\n\n\n<div class=\"wp-block-column\" style=\"flex-basis:33.33%\">\n(1)<\/p>\n<\/div>\n<\/div>\n\n\n\nwhere Treati<\/sub> is a treatment indicator that takes a value of 1 if unit i belongs to the treatment group and 0 if it belongs to the control group, Postt<\/sub> is a posttreatment time period indicator that takes a value of 1 if time period t is in the post-treatment time period and 0 otherwise, xi<\/sub> is a k-dimensional vector of time invariant observable covariates, \u03b2\u02dc = (\u03b25<\/sub>, ..., \u03b25 + k<\/sub>)\u2032, and \u03f5it<\/sub> is an error term. The coefficient \u03b24<\/sub> is the causal effect of interest, which is the average treatment effect (ATE) or average treatment effect on the treated (ATT).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-assumptions\">Assumptions<\/em><\/h3>\n\n\n\nBefore choosing the DID identification strategy, researchers first need to assess whether the identifying parallel trends assumption holds. The DID parallel trends assumption states that the treatment unit would have followed a path parallel to the control units in the absence of treatment. We make two observations about this assumption. First, the DID method can be interpreted as a method that primarily matches on outcomes. Although covariates can be included in the DID regression model, the main goal is to use the control units\u2019 outcomes to match the treatment unit\u2019s outcome during the pretreatment period, and then predict the treatment counterfactual and the ATT. Second, since the parallel trends assumption is a statement about the treatment counterfactual, we cannot directly test the parallel trends assumption. However, what we can do is test whether the treatment and control units followed parallel trends in the pretreatment period (parallel pretrends assumption). This is essentially the testable part of the parallel trends assumption. There are two popular approaches to check the testable part of the parallel trends assumption: (1) visual inspection and (2) statistical tests. Visual inspection involves plotting the treatment and control trends in the pretreatment period and inspecting whether they look parallel. Statistical tests \u201cformalize\u201d this evaluation somewhat, by testing whether the difference in mean outcomes of the treatment and control group, for each pretreatment period, are statistically different from a constant. However, statistical tests often have low power (i.e., they may fail to reject the null hypothesis of no difference even when the parallel pretrends assumption is violated), implying that visual inspection can be the only viable way of assessing whether the parallel pretrends assumption holds.<\/p>\n\n\n\nIf the parallel pretrends assumption of the DID method is violated, researchers can attempt to use matching methods (described in more detail in the \u201cSelection on Variables\u201d section ) to first identify a subset of control units that are more similar to the treatment unit on covariates and then apply the DID method. Alternatively, if the number of treatment units is not very large (e.g., less than 100), researchers can apply the synthetic control or related methods (described in more detail in the \u201cSynthetic Control and Related Methods\u201d section).<\/p>\n\n\n\nAnother assumption underlying the use of DID methods (and many causal inference methods) is the stable unit treatment value assumption (SUTVA). SUTVA can be decomposed into two parts. The first part of SUTVA (no interference) requires that treatment applied to one unit does not affect the outcome of other units. To understand this better, let us consider the case where a state experiences a treatment (e.g., enactment of a local tax law). This should mean that the treatment should not affect outcomes in the control states or other treatment states (or vice versa). Researchers can use logical argumentation based on institutional knowledge to justify this assumption (e.g., geographical variation in treatment and control units makes interference unlikely). To the extent that researchers have access to more granular data, they can also check and confirm that there is no movement of individuals across treatment and control states, patterns that support the case for no interference. The second part of SUTVA (no hidden variations of treatment) requires that for each unit, there are no different forms or versions of each treatment level that may lead to different potential outcomes. Researchers can use institutional knowledge about the treatment itself as justification for this assumption.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-staggered-treatment-timing\">Staggered Treatment Timing<\/em><\/h3>\n\n\n\nAs researchers, we are commonly faced with treatments that apply to different units at different times. This is referred to as differential timing of treatment or staggered treatment timing. When treatment effects are homogeneous over time, the following regression equation can be used to estimate the DID model:<\/p>\n\n\n\n<div class=\"wp-block-columns is-not-stacked-on-mobile\">\n<div class=\"wp-block-column\" style=\"flex-basis:66.66%\">\nyit<\/sub> = \u03b21<\/sub> + \u03b22<\/sub>Treati<\/sub>Postt<\/sub> + xi<\/sub>\u03b2\u02dc + FEi<\/sub> + FEt<\/sub> + \u03f5it<\/sub>.<\/p>\n<\/div>\n\n\n\n<div class=\"wp-block-column\" style=\"flex-basis:33.33%\">\n(2)<\/p>\n<\/div>\n<\/div>\n\n\n\nThis model is called a two-way fixed effects (TWFE) model because it contains fixed effects for both unit (FEi<\/sub>) and time (FEt<\/sub>). However, the homogeneity in treatment response can be a restrictive assumption, as it is common that the treatment effect may change over time or differ across treatment units.<\/p>\n\n\n\n<a href=https://www.ama.org/"https:////doi.org//10.1016//j.jeconom.2021.03.014/" target=\"_blank\">Goodman-Bacon (2021)<\/a> identified that for the case of heterogeneous treatment effects and staggered treatment timing, the conventional TWFE model breaks down (i.e., yields a biased average treatment effect). The intuition is that the conventional TWFE model uses a weighted average of all potential DID estimates obtained using different combinations of groups of treatment and control units (treated, not yet treated, and never treated groups). However, a group that has already been treated can only be used as a control group for a group that is treated later, in the case of time-invariant treatment effects. In other words, the already treated group is not a \u201cclean\u201d control when the treatment effect varies over time (we also refer to this as heterogeneous or time-variant treatment effects). Because the standard TWFE model uses control units that are not \u201cclean,\u201d it is biased when the treatment effect varies over time. While Goodman-Bacon (2021) helps the researcher identify the specific sources of bias in their setting by decomposing the standard DID estimator into different underlying comparisons (e.g., early vs. late treated, treated vs. never treated), this paper does not provide a single, unbiased estimate of the treatment effect.<\/p>\n\n\n\nTo solve the problem identified in Goodman-Bacon (2021), scholars have proposed many estimators. We discuss three proposed solutions. The first solution is an estimator proposed by <a href=https://www.ama.org/"https:////doi.org//10.1016//j.jeconom.2020.12.001/" target=\"_blank\">Callaway and Sant\u2019Anna (2021)<\/a>. It is perhaps the most popular and widely used. The main idea is to first estimate the ATT for each treatment group cohort and then use a weighted average of those ATTs. Specifically, ATT(g) is the ATT for the treatment group cohort that first receives treatment at time period g. To estimate ATT(g), we can use a simple DID setup, where the treatment group is the treatment group cohort g and the control group are units that are not yet treated or never treated (excluding units that are already treated). Then, the overall ATT is a weighted average of all the ATT(g)\u2019s estimated over different time periods. Callaway and Sant\u2019Anna (2021) can accommodate an outcome-regression-based estimator (<a href=https://www.ama.org/"https:////doi.org//10.2307//2971733/" target=\"_blank\">Heckman, Ichimura, and Todd 1997<\/a>), an inverse probability weighted estimator (<a href=https://www.ama.org/"https:////doi.org//10.1111//0034-6527.00321/" target=\"_blank\">Abadie 2005<\/a>), and a doubly robust estimator (<a href=https://www.ama.org/"https:////doi.org//10.1016//j.jeconom.2020.06.003/" target=\"_blank\">Sant\u2019Anna and Zhao 2020<\/a>).<\/p>\n\n\n\nThe second solution, which is closely related to Callaway and Sant\u2019Anna (2021), is an estimator proposed by <a href=https://www.ama.org/"https:////doi.org//10.1016//j.jeconom.2020.09.006/" target=\"_blank\">Sun and Abraham (2021)<\/a>. This estimator is an extension of Callaway and Sant\u2019Anna (2021) to the case of dynamic treatment effects, where researchers are interested in estimating separate treatment effects for each posttreatment time period. This setting experiences the same problem that already treated units are not \u201cclean\u201d controls when the treatment effect is heterogeneous. If researchers need to calculate a treatment effect estimate for every posttreatment period, then they should use Sun and Abraham (2021).<\/p>\n\n\n\nThe third solution is the \u201cstacked regression\u201d proposed by applied researchers (<a href=https://www.ama.org/"https:////doi.org//10.1093//qje//qjz014/" target=\"_blank\">Cengiz et al. 2019<\/a>; <a href=https://www.ama.org/"https:////doi.org//10.1093//rfs//hhr011/" target=\"_blank\">Gormley and Matsa 2011<\/a>). The main idea is to create separate clean datasets of treatment groups and \u201cclean\u201d control groups, stack them by aligning the intervention time period, and use the DID TWFE regression with dataset and time fixed effects. While in practice, this solution is the simplest to implement, one may need to be cautious, as the sample average ATT may be inconsistent (<a href=https://www.ama.org/"https:////doi.org//10.1016//j.jfineco.2022.01.004/" target=\"_blank\">Baker, Larcker, and Wang 2022<\/a>).<\/p>\n\n\n\nGiven the discussions above, we provide the following recommendations. First, if all treatment units receive treatment at the same time (no staggered timing), use the standard DID TWFE model. However, if there is staggered treatment timing, use one of the proposed solutions and justify the choice of clean controls. Finally, as a robustness check, researchers can separately estimate a DID TWFE for each treatment cohort group using clean controls (e.g., the never treated group) (Baker, Larcker, and Wang 2022). Note that each of these separate analyses does not suffer from the issues that affect staggered DID with heterogeneous treatment effects because each analysis only examines one treatment cohort at a time (no staggered treatment timing). While this robustness check does not result in an aggregated treatment effect, it may still be informative to show what the separate ATTs are for each treatment group cohort.<a href=https://www.ama.org/"#ftn1\">[1]<\/a><\/sup><\/p>\n\n\n\nFor a more detailed discussion of these three solutions among many others, we refer readers to two review papers: \u201cWhat\u2019s Trending in Difference-in-Differences? A Synthesis of the Recent Econometric Literature\u201d by <a href=https://www.ama.org/"https:////doi.org//10.1016//j.jeconom.2023.03.008/" target=\"_blank\">Roth et al. (2023)<\/a> and \u201cHow Much Should We Trust Staggered Difference-in-Differences Estimates?\u201d by Baker, Larcker, and Wang (2022). Note that all of the methods discussed in these two articles deal with settings characterized by a large number of treatment and control units and relatively short time periods.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-synthetic-control-and-related-methods\"><a><\/a>Synthetic Control and Related Methods<\/h2>\n\n\n\nWhile the DID method is the most popular quasi-experimental method, it is often not viable, due to obvious observed violations in the parallel pretrends assumption. To overcome this, there has been a recent surge in flexible alternative estimators that are more widely applicable than DID. The most famous of such methods is the synthetic control method proposed by <a href=https://www.ama.org/"https:////www.aeaweb.org//articles?id=10.1257\/000282803321455188\%22 target=\"_blank\">Abadie and Gardeazabal (2003)<\/a> and <a href=https://www.ama.org/"https:////doi.org//10.1198//jasa.2009.ap08746/" target=\"_blank\">Abadie et al. (2010)<\/a>. The synthetic control method has been called \u201carguably the most important innovation in the evaluation literature in the last fifteen years\u201d by Susan Athey and Guido Imbens (the latter is winner of the 2021 Nobel Prize in Economics for making \u201cmethodological contributions to the analysis of causal relationships\u201d).<\/p>\n\n\n\nThe synthetic control method uses a weighted average of the control units (instead of a simple average used in the DID) to predict the treatment counterfactual and the ATT. The synthetic control method, like DID, primarily matches on outcomes. It achieves this by using a weighted average of control units\u2019 outcomes to match the treatment unit\u2019s outcomes during the pre-treatment period. This approach better matches the treatment outcomes in the pre- treatment period, which then improves the prediction of the counterfactual and consequently, the estimate of the average treatment effect on the treated (ATT). However, the guidance for conducting proper inference using the synthetic control method is not clear. Previously, researchers had to rely on placebo tests, which entail making restrictive assumptions that are often violated. <a href=https://www.ama.org/"https:////doi.org//10.1080//01621459.2019.1686986/" target=\"_blank\">Li (2020)<\/a> developed the inference theory for the synthetic control method, which allows the calculation of confidence intervals and quantifying uncertainty using a subsampling procedure.<\/p>\n\n\n\nWhile the synthetic control method is a very powerful new tool, it still has some associated restrictions. First, it is unable to easily accommodate settings involving a large number of treated units. Second, it is less suited for handling situations wherein the treatment and control units are very different from one another (e.g., situations where the outcome for the treatment unit is outside the range of that for the control units). Such settings call for more flexible methods, the most popular of which is the factor model.<\/p>\n\n\n\nIn many marketing contexts, researchers have access to a large number of control units. The factor model in particular has gained traction in marketing due to its ability to elegantly handle such situations via dimension reduction. Conveniently, the dimension reduction of the factor model also serves as implicit regularization to prevent overfitting. The factor model is also known as generalized synthetic control (<a href=https://www.ama.org/"https:////doi.org//10.1017//pan.2016.2/" target=\"_blank\">Xu 2017<\/a>) or interactive fixed effects model (<a href=https://www.ama.org/"https:////ideas.repec.org//p//syd//wpaper//2016-11.html/">Chan and Kwok 2016<\/a>; <a href=https://www.ama.org/"https:////www.jstor.org//stable//24917033/" target=\"_blank\">Gobillon and Magnac 2016<\/a>).<\/p>\n\n\n\nMarketing researchers have applied this method in a variety of settings ranging from policy evaluation (<a href=https://www.ama.org/"https:////doi.org//10.1287//mksc.2019.1181/" target=\"_blank\">Guo, Sriram, and Manchanda 2020<\/a>; <a href=https://www.ama.org/"https:////doi.org//10.1177//0022242918815163/" target=\"_blank\">Pattabhiramaiah, Sriram, and Manchanda 2019<\/a>) to advertising effects measurement (<a href=https://www.ama.org/"https:////link.springer.com//article//10.1007//s11129-019-09211-9/" target=\"_blank\">Lovett, Peres, and Xu 2019<\/a>). How should researchers quantify uncertainty when using the synthetic control in combination with the factor model? Past research has recommended the use of a bootstrap procedure (Xu 2017), which can be restrictive. <a href=https://www.ama.org/"https:////doi.org//10.1177//00222437221137533/" target=\"_blank\">Li and Sonnier (2023)<\/a> show that the bootstrap procedure provided in Xu (2017) often results in biased confidence intervals that are either too narrow or too wide, leading to false precision or false imprecision. False precision may lead researchers to erroneously conclude that they detected a true effect, whereas false imprecision may lead researchers to erroneously conclude that there was no detectable true effect. Both mistakes\u2014false positives and false negatives\u2014can lead to incorrect business decisions. Following the inference theory in Li and Sonnier (2023), researchers can correctly quantify uncertainty of causal effects to make more informed business decisions.<\/p>\n\n\n\nThe synthetic DID method proposed by <a href=https://www.ama.org/"https:////www.aeaweb.org//articles?id=10.1257\/aer.20190159\%22 target=\"_blank\">Arkhangelsky et al. (2021)<\/a> is another flexible quasi-experimental method that has gained traction in marketing. Marketing scholars have used synthetic DID to study the effect of TV advertising on online browsing and sales (<a href=https://www.ama.org/"https:////doi.org//10.1177//00222437231180171/" target=\"_blank\">Lambrecht, Tucker, and Zhang 2024<\/a>) and the effect of soda tax on marketing effectiveness (<a href=https://www.ama.org/"https:////doi.org//10.1177//00222437231195551/" target=\"_blank\">Keller, Guyts, and Grewal 2024<\/a>). The synthetic DID method proposes a general framework that uses both individual weights and time weights for additional flexibility. To conduct inference and compute standard errors, Arkhangelsky et al. offer three alternative procedures: (1) block bootstrap, (2) jackknife, and (3) permutation. Block bootstrap and jackknife require a large number of treatment units, without which the estimated confidence intervals may be unreliable (Clarke et al. forthcoming). Permutation does not have any restriction on the number of treatment units, but it requires a moderate to large number of control units and requires that the treatment unit and control units\u2019 variance be similar.<\/p>\n\n\n\nNext, we overview two additional quasi-experimental methods. The first is the ordinary least squares (OLS) method proposed by <a href=https://www.ama.org/"https:////doi.org//10.1002//jae.1230/" target=\"_blank\">Hsiao, Ching, and Wan (2012)<\/a> (which is also called the HCW method). The OLS method can be used when the number of control units is (much) smaller than the pretreatment time periods. Due to the increased flexibility of both the OLS method and synthetic DID methods, it is even more important when using these methods to check for overfitting using the backdating exercise described the last paragraph of this section. Another method is the matrix completion method (for additional details, see <a href=https://www.ama.org/"https:////doi.org//10.1146//annurev-economics-080217-053433/" target=\"_blank\">Athey and Imbens [2019]<\/a> and <a href=https://www.ama.org/"https:////doi.org//10.1080//01621459.2021.1967163/" target=\"_blank\">Bai and Ng [2021]<\/a>). The matrix completion method imputes missing values in a panel data setup to estimate counterfactuals when potential outcomes have a factor structure to estimate the ATT. In other words, this approach estimates missing values in a dataset by using principal components analysis, which can be used to estimate the effects of a treatment when some of the potential outcomes are missing (for a recent marketing application of this method, see <a href=https://www.ama.org/"https:////doi.org//10.1287//mksc.2019.1189/" target=\"_blank\">Bronnenberg, Dub\u00b4e, and Sanders [2020]<\/a>).<\/p>\n\n\n\nAll the methods described so far are frequentist methods. However, each of these methods can be estimated using a Bayesian framework. <a href=https://www.ama.org/"https:////doi.org//10.1177//0022243720936230/" target=\"_blank\">Kim, Lee, and Gupta (2020)<\/a> propose a Bayesian synthetic control method that uses Bayesian shrinkage priors to solve the sparsity problem and conduct inference. <a href=https://www.ama.org/"https:////doi.org//10.1017//pan.2021.22/" target=\"_blank\">Pang, Liu, and Xu (2022)<\/a> propose a Bayesian factor model that uses a Bayesian shrinkage method for model searching and factor selection.<\/p>\n\n\n\nThe synthetic control method and related flexible alternative methods (e.g. factor model method, synthetic DID method, OLS method, matrix competition method) we have discussed thus far require access to a sufficiently long pre-treatment time window (e.g. at least ten pretreatment time periods). However, what if researchers need a flexible alternative to DID but do not have access to a sufficient number of time periods before the treatment occurs? To fill this gap, researchers can consider using the augmented DID or forward DID methods. Specifically, if the outcome for the treatment unit is outside of the range of that of the control units, researchers can use the augmented DID method, which uses a scaled average of the control units to construct the treatment counterfactual (<a href=https://www.ama.org/"https:////doi.org//10.1287//mksc.2022.1406/" target=\"_blank\">Li and Van den Bulte 2023<\/a>). On the other hand, if the outcome for treatment unit is within the range of that of the control units, researchers can consider using the forward DID method, which uses a forward selection algorithm to select a relevant subset of control units and then applies the DID method (<a href=https://www.ama.org/"https:////doi.org//10.1287//mksc.2022.0212/" target=\"_blank\">Li 2024<\/a>).<\/p>\n\n\n\nBuilding on recent advances in the literature studying flexible alternatives to DID, we recommend following two best practices when implementing the synthetic control and related methods. First, after applying the method, visually inspect whether the parallel trends assumption of the corresponding method holds in the pretreatment window by plotting the outcome variable corresponding to the treatment unit(s) and that of the fitted in-sample curve, which is created using the control units. If the parallel pretrends assumption does not hold, do not adopt the method. If the parallel pretrends assumption does hold, then continue to conduct a backdating (out-of-sample prediction) exercise to check for overfitting (<a href=https://www.ama.org/"https:////doi.org//10.1257//jel.20191450/" target=\"_blank\">Abadie 2021<\/a>; Li 2020; Li and Sonnier 2023). We recommend only using the methods that satisfy both best practices.<\/p>\n\n\n\nOur discussion thus far has focused on methods aimed primarily at matching treated and control units on the outcome variable (Y). In the next section, we discuss approaches focused on matching on observable covariates (X).<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-selection-on-observables-and-consequently-on-unobservables\">Selection on Observables (and Consequently, on Unobservables)<\/h2>\n\n\n\nAs we have noted, one main challenge in estimating causal effects from observational studies is the presence of confounding factors that simultaneously affect the treatment status and the outcome of interest. In some observational studies, the allocation of treatment can be presumed to resemble random assignment (e.g., a policy change determined independently from the outcomes being measured). However, such situations are rare because forces such as consumer incentives, firm objectives, or regulation can threaten such a pure treatment exogeneity argument. For such reasons, researchers often find themselves in two states of the world\u2014one where covariates are not available, and one where they are. In situations where suitable covariates may not be available, researchers may consider methods such as instruments, copulas, and control function to infer causality using observational data. We refer readers to Wooldridge (2019), <a href=https://www.ama.org/"https:////doi.org//10.1509//jmkr.47.1.3/" target=\"_blank\">Petrin and Train (2010)<\/a>, <a href=https://www.ama.org/"https:////doi.org//10.1287//mksc.1120.0718/" target=\"_blank\">Park and Gupta (2012)<\/a>, and <a href=https://www.ama.org/"https:////doi.org//10.1287//mksc.1090.0491/" target=\"_blank\">Danaher and Smith (2010)<\/a> for additional details.<\/p>\n\n\n\nHowever, in our information-rich era, researchers generally have abundant covariates at hand. Oftentimes, researchers observe many\/most confounding factors that could simultaneously affect the treatment status and the outcome of interest. For example, <a href=https://www.ama.org/"https:////doi.org//10.1287//mksc.2022.1401/" target=\"_blank\">Ellickson, Kar, and Reeder (2023)<\/a> consider the case of observational studies using data from targeted marketing campaigns, where, following the definition of the targeting rule, the treatment assignment is determined based on some observable demographic or behavioral covariates. Thus, there are many observational studies where the unconfoundedness (or selection on observables) assumption is satisfied, and researchers can adopt methods to adjust for the observed confounders.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-matching-on-covariates\"><a><\/a>Matching on Covariates<\/em><\/h3>\n\n\n\nMatching methods, in simple terms, aim to pair\/match units with similar covariates but different treatment statuses to estimate the treatment effects by comparing their outcomes. Some well-known traditional methods aimed at adjusting for observed confounders are parametric matching, propensity scores, and weighting methods. All these methods require that the researcher has knowledge about which covariates are important a priori. The identification of the observed confounders and the selection of the variables that represent them is usually based on economic theories, institutional knowledge, or intuition (e.g., targeting ads depends on consumer engagement on a website).<\/p>\n\n\n\nPropensity score matching has been a commonly used matching method in marketing for decades, although their viability has recently been called into question due to the technique\u2019s sensitivity to parametric assumptions (<a href=https://www.ama.org/"https:////www.aeaweb.org//articles?id=10.1257\/jep.31.2.3\%22 target=\"_blank\">Athey and Imbens 2017<\/a>). These methods start with the estimation of the propensity score (i.e., the probability of receiving the treatment conditional on covariates (ei<\/sub> = P(Ti<\/sub> = 1|Xi<\/sub>)), which can be later combined with matching, stratification, inverse probability weighting, or covariate adjustment (<a href=https://www.ama.org/"https:////doi.org//10.1080//00273171.2011.568786/" target=\"_blank\">Austin 2011<\/a>). Another popular method to estimate treatment effects under the unconfoundedness assumption is the augmented inverse probability weighting approach (<a href=https://www.ama.org/"https:////doi.org//10.2307//2290910/" target=\"_blank\">Robins, Rotnitzky, and Zhao 1994<\/a>), which combines regression models to estimate the potential outcomes with inverse propensity score weighting methods. One attractive property of this estimator is its robustness to bias or misspecifications in either the potential outcome estimate or the propensity score estimate (<a href=https://www.ama.org/"https:////doi.org//10.1111//j.1541-0420.2005.00377.x/" target=\"_blank\">Bang and Robins 2005<\/a>). See<a href=https://www.ama.org/"https:////doi.org//10.1287//mksc.2018.1135/" target=\"_blank\"> Gordon et al. (2019)<\/a> for a recent example in marketing that illustrates such applications in the context of online advertising.<\/p>\n\n\n\nDespite their popularity, matching methods have limitations, especially when the number of covariates is very large. In such cases, conventional methods such as exact-matching might become infeasible, and nearest-neighbor matching can result in a biased estimate of the ATT (<a href=https://www.ama.org/"https:////doi.org//10.1111//j.1468-0262.2006.00655.x/" target=\"_blank\">Abadie and Imbens 2006<\/a>). To obtain a flexible specification of the propensity score when the number of covariates is large, we can apply variable reduction methods such as Lasso (e.g., Gordon et al. 2019) or penalized logistic regression (e.g., <a href=https://www.ama.org/"https:////doi.org//10.1080//01621459.2020.1796393/" target=\"_blank\">Eckles and Bakshy 2021<\/a>). Additionally, we can use machine learning (ML) methods to reduce the dimension of the covariate space. For example, <a href=https://www.ama.org/"https:////dl.acm.org//doi//abs//10.5555//3061053.3061146/" target=\"_blank\">Li et al. (2016)<\/a> illustrate the usefulness of using linear dimensionality reduction ML algorithms such as principal component analysis (PCA), locality preserving projections (LPP), and random projections before matching the treatment and control units. <a href=https://www.ama.org/"https:////doi.org//10.48550//arXiv.1803.00149/" target=\"_blank\">Ramachandra (2018)<\/a> explores the use of auto-encoders as a dimensionality reduction technique prior to neighbor matching on simulated data. In a similar vein, <a href=https://www.ama.org/"https:////dl.acm.org//doi//10.5555//3327144.3327188/" target=\"_blank\">Yao et al. (2018)<\/a> develop a method based on deep representation learning that jointly preserves the local similarity information and balances the distributions of the control and the treated groups.<\/p>\n\n\n\nAdditionally, <a href=https://www.ama.org/"https:////doi.org//10.1162//REST_a_00318/" target=\"_blank\">Diamond and Sekhon (2013)<\/a> propose GenMatch, a multivariate matching method based on genetic algorithm to iteratively check and improve covariate balance between the treated and the control groups. <a href=https://www.ama.org/"https:////doi.org//10.1080//01621459.2015.1023805/" target=\"_blank\">Zubizarreta (2015)<\/a> also proposes a weighting method that allows researchers to prespecify the level of desired balance between the treated and the control groups. One advantage of this weighting method is that it runs in polynomial time, so large datasets can be handled quickly.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-causal-machine-learning-methods-on-flexible-matching\">Causal Machine Learning Methods on Flexible Matching<\/em><\/h3>\n\n\n\nWhile matching on covariates can be highly powerful in many research settings, the methods discussed in the previous section often require that the researchers have prior knowledge about which covariates are important and which functional form is the most suitable for capturing their influence on the outcome variables. However, when working in high-dimensional settings, it might become difficult for researchers to identify which specific covariates are important (e.g., number of clicks, time spent on different sections of the website) or which functional form is appropriate for modeling their influence on outcomes (linear, quadratic, or more flexible specifications). Meanwhile, including all the covariates or allowing for flexible functional forms may reduce the power available in the dataset for learning about the treatment effect of interest (<a href=https://www.ama.org/"https:////doi.org//10.1111//ectj.12097/" target=\"_blank\">Chernozhukov et al. 2018<\/a>). In such cases, researchers can benefit from adopting causal ML methods for flexible matching. Next, we discuss some commonly used causal ML methods often used on observational data for inferring causality. These methods are especially helpful in settings that involve high-dimensional covariates and\/or when the relationship between them cannot be satisfactorily modeled in a parametric way. In such cases, ML methods will arguably provide a better specification of the propensity score and outcome models than more traditional methods.<\/p>\n\n\n\nOne such example is the doubly robust estimator (<a href=https://www.ama.org/"https:////doi.org//10.1111//j.1541-0420.2005.00377.x/">Bang and Robins 2005<\/a>) that leverages ML methods for predicting both the propensity score and the potential outcome variables. The method then allows a doubly robust estimation of the potential outcome preserving the favorable statistical properties that permit rigorous causal inference. Another recent development in the estimation of treatment effects under the unconfoundedness assumption is the use of ML methods to directly make inference about the parameters using the double ML approach (<a href=https://www.ama.org/"https:////doi.org//10.1111//ectj.12097/">Chernozhukov et al. 2018<\/a>). This method involves using ML methods to residualize any potential impact that the covariates may have had on both the treatment and the outcome variables. The double ML framework can be combined with doubly robust estimators. It can also be readily extended to estimate heterogeneous treatment effects.<\/p>\n\n\n\nWe can also use tree-based ML approaches such as causal forest (<a href=https://www.ama.org/"https:////doi.org//10.1080//01621459.2017.1319839/" target=\"_blank\">Wager and Athey 2018<\/a>) for estimating both average and heterogeneous treatment effects in observational studies where the unconfoundedness assumption is satisfied. As discussed in <a href=https://www.ama.org/"https:////www.aeaweb.org//articles?id=10.1257\/jep.31.2.3\%22 target=\"_blank\">Athey and Imbens (2016)<\/a> and Wager and Athey (2018), the causal forest model can be particularly suitable for inferring treatment effects from rich observational data containing a large number of covariates. In contrast with conventional propensity score matching, causal forests utilize a flexible nonparametric data-driven approach to determine similarity across observations. Additionally, the estimation of traditional propensity score methods is often sensitive to the model specification (<a href=https://www.ama.org/"https:////projecteuclid.org//journals//annals-of-applied-statistics//volume-12//issue-1//Covariate-balancing-propensity-score-for-a-continuous-treatment--Application//10.1214//17-AOAS1101.full/" target=\"_blank\">Fong, Hazlett, and Imai 2018<\/a>), especially when the treatment variable is continuous. The causal forests are immune to such problems because the building of an honest tree (the building block of causal forests) does not rely on any particular functional form. Some recent examples in marketing of the use of causal forests for inferring causalit<a href=https://www.ama.org/"https:////doi.org//10.1287//mksc.2022.1401/">y from observational data are Guo, Sriram, and Manchanda (2021),<\/a> Ellickson, Kar, and Reeder (2023), <a href=https://www.ama.org/"https:////doi.org//10.1287//mnsc.2021.4092/" target=\"_blank\">Pattabhiramaiah, Overby, and Xu (2022)<\/a>, and <a href=https://www.ama.org/"https:////doi.org//10.1287//mnsc.2022.4359/" target=\"_blank\">Zhang and Luo (2023)<\/a>.<\/p>\n\n\n\nFor any of the quasi-experimental methods discussed above, researchers can also conduct sensitivity analyses, which involve assessing the extent of unobserved confounding necessary for nullifying the causal effect (<a href=https://www.ama.org/"https:////www.aeaweb.org//articles?id=10.1257\/aer.98.2.345\%22 target=\"_blank\">Altonji, Elder, and Taber 2008<\/a>; <a href=https://www.ama.org/"https:////www.aeaweb.org//articles?id=10.1257\/000282803321946921\%22 target=\"_blank\">Imbens 2003<\/a>; <a href=https://www.ama.org/"https:////www.jstor.org//stable//2345524/" target=\"_blank\">Rosenbaum and Rubin 1983<\/a>). <a href=https://www.ama.org/"https:////doi.org//10.1007//s11121-012-0339-5/" target=\"_blank\">Liu, Kuramoto, and Stuart (2013)<\/a> provide a nice introduction to sensitivity analysis. Recent work (e.g., <a href=https://www.ama.org/"https:////doi.org//10.1111//rssb.12348/" target=\"_blank\">Cinelli and Hazlett 2020<\/a>; <a href=https://www.ama.org/"https:////doi.org//10.1080//07350015.2016.1227711/" target=\"_blank\">Oster 2019<\/a>) has expanded on this idea to formally bound the strength of unobserved confounders by comparing them with observed covariates. Oster (2019) argues that the robustness of estimates to omitted variable bias can be examined by observing movements in (1) the coefficient of interest and (2) model R2<\/sup> from specifications that either include or exclude control variables in a regression. <a href=https://www.ama.org/"https:////doi.org//10.48550//arXiv.2208.00552/" target=\"_blank\">Masten and Poirier (2022)<\/a> point out that unobserved confounders can either drive baseline estimates to zero or reverse their sign, with the latter actually being easier. They recommend several best practices for sensitivity assessment and even offer a companion Stata package to help researchers adopt these tools.<\/p>\n\n\n\nLast but not least, in addition to using sophisticated statistical\/econometric\/ML methods for mitigating such concerns, we can also consider using field or lab experiments to complement causal conclusions drawn from field data, especially for forming a deeper understanding of the underlying mechanisms (for some recent examples, see <a href=https://www.ama.org/"https:////doi.org//10.1177//00222429211044155/" target=\"_blank\">Nickerson et al. [2023]<\/a> and <a href=https://www.ama.org/"https:////doi.org//10.1177//00222437231220327/" target=\"_blank\">Anderson et al. [2024]<\/a>).<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-conclusion\">Conclusion<\/h2>\n\n\n\nThe main purpose of this article is to offer some guidance to help marketing researchers choose the most appropriate method for understanding causal relationships from quasi-experimental data. We begin with the basic DID method that is widely used in settings with treatment and control units and pre- and posttreatment periods. We thereafter discuss advances in using DID for contexts characterized by staggered treatment timing and heterogeneous treatment effects. We then explore flexible alternatives to DID, such as the synthetic control method, which is predicated on the researcher\u2019s access to a relatively large number of pre-treatment periods, and other alternative methods that do not require a large number of pretreatment periods. We cover how to estimate causal effects, conduct inference, and recommend best practices for these alternatives. Additionally, we review quasi-experimental methods with covariates, such as matching, and recent advances in causal ML methods for flexible matching. Given the rapid development of new methods, this article is not meant to be an exhaustive review of the literature on causal inference using observational data, but rather a useful starting point. Recent research has introduced novel ways for combining ML with instrumental variables (e.g., <a href=https://www.ama.org/"https:////proceedings.mlr.press//v70//hartford17a//hartford17a.pdf/" target=\"_blank\">Hartford et al. 2017<\/a>; <a href=https://www.ama.org/"https:////doi.org//10.1145//3391403.3399466/" target=\"_blank\">Singh, Hosanagar, and Gandhi 2020<\/a>) and incorporating natural language processing techniques within a causal framework (<a href=https://www.ama.org/"https:////doi.org//10.1162//tacl_a_00511/" target=\"_blank\">Feder et al. 2022<\/a>) with the goal of improving causal inference. Thus, the researcher\u2019s methodological toolkit is ever expanding. We hope that this writeup helps guide researchers identify the right set of tools for answering causal research questions based on the data characteristics of their problem.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"h-references\">References<\/h4>\n\n\n\nAbadie, Alberto (2005), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1111//0034-6527.00321/" target=\"_blank\"><\/a><a href=https://www.ama.org/"https:////www.jstor.org//stable//3700681/" target=\"_blank\" rel=\"noreferrer noopener\">Semiparametric Difference-in-Differences Estimators<\/a>,\u201d Review of Economic Studies<\/em>, 72 (1), 1\u201319.<\/p>\n\n\n\nAbadie, Alberto (2021), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1257//jel.20191450/" target=\"_blank\">Using Synthetic Controls: Feasibility, Data Requirements, and Methodological Aspects<\/a>,\" Journal of Economic Literature<\/em>, 59 (2), 391\u2013425.<a id=\"_bookmark8\"><\/a><\/p>\n\n\n\nAbadie, Alberto, Alexus Diamond, and Jens Hainmueller (2010), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1198//jasa.2009.ap08746/" target=\"_blank\">Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California\u2019s Tobacco Control Program<\/a>,\u201d Journal of the American Statistical Association<\/em>, 105 (490), 493\u2013505.<a id=\"_bookmark9\"><\/a><\/p>\n\n\n\nAbadie, Alberto and Javier Gardeazabal (2003), \u201c<a href=https://www.ama.org/"https:////www.aeaweb.org//articles?id=10.1257\/000282803321455188\%22 target=\"_blank\">The Economic Costs of Conflict: A Case Study of the Basque Country<\/a>.\" American Economic Review<\/em>, 93 (1), 113\u201332.<a id=\"_bookmark10\"><\/a><\/p>\n\n\n\nAbadie, Alberto and Guido W. Imbens (2006), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1111//j.1468-0262.2006.00655.x/" target=\"_blank\">Large Sample Properties of Matching Estimators for Average Treatment Effects<\/a>,\u201d Econometrica<\/em>, 74 (1), 235\u201367.<\/p>\n\n\n\nAltonji, Joseph G., Todd E. Elder, and Christopher R. Taber (2008), <a href=https://www.ama.org/"https:////www.aeaweb.org//articles?id=10.1257\/aer.98.2.345\%22 target=\"_blank\">\u201cUsing Selection on Observed Variables to Assess Bias from Unobservables When Evaluating Swan-Ganz Catheterization<\/a>,\u201d American Economic Review <\/em>, 98 (2), 345\u201350.<\/p>\n\n\n\nAnderson, Eric, Chaoqun Chen, Ayelet Israeli, and Duncan Simester (2024), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1177//00222437231220327/" target=\"_blank\">Canary Categories<\/a>,\u201d Journal of Marketing <a id=\"_bookmark13\"><\/a>Research<\/em>, 61 (5), 872\u201390.<\/p>\n\n\n\nArkhangelsky, Dmitry, Susan Athey, David A. Hirshberg, Guido W. Imbens, and Stefan Wager (2021), \u201c<a href=https://www.ama.org/"https:////www.aeaweb.org//articles?id=10.1257\/aer.20190159\%22 target=\"_blank\">Synthetic Difference-in-Differences<\/a>,\u201d American Economic Review <\/em>, 111 (12), 4088\u20134118.<a id=\"_bookmark14\"><\/a><\/p>\n\n\n\nAthey, Susan and Guido Imbens (2017), \u201c<a href=https://www.ama.org/"https:////www.aeaweb.org//articles?id=10.1257\/jep.31.2.3\%22 target=\"_blank\">The State of Applied Econometrics: Causality and Policy Evaluation<\/a>,\u201d Journal of Economic Perspectives<\/em>, 31 (2), 3\u201332.<\/p>\n\n\n\nAthey, Susan and Guido W. Imbens (2019), <a href=https://www.ama.org/"https:////doi.org//10.1146//annurev-economics-080217-053433/" target=\"_blank\" rel=\"noreferrer noopener\">\u201cMachine Learning Methods That Economists Should Know About<\/a>,\u201d Annual Review of Economics<\/em>, 11, 685\u2013725.<\/p>\n\n\n\nAustin, Peter C. (2011), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1080//00273171.2011.568786/" target=\"_blank\">An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies<\/a>,\u201d Multivariate Behavioral Research<\/em>, 46 (3), 399\u2013424.<\/p>\n\n\n\nBai, Jushan and Serena Ng (2021), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1080//01621459.2021.1967163/" target=\"_blank\" rel=\"noreferrer noopener\">Matrix Completion, Counterfactuals, and Factor Analysis of Missing Data<\/a>,\u201d Journal of the American Statistical Association<\/em>, 116 (536), 1746\u201363.<\/p>\n\n\n\nBaker, Andrew C., David F. Larcker, and Charles C.Y. Wang (2022), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1016//j.jfineco.2022.01.004/" target=\"_blank\" rel=\"noreferrer noopener\">How Much Should We Trust Staggered Difference-In-Differences Estimates?<\/a>\u201d Journal of Financial Economics<\/em>, 144 (2), 370\u201395.<\/p>\n\n\n\nBang, Heejung and James M. Robins (2005), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1111//j.1541-0420.2005.00377.x/" target=\"_blank\" rel=\"noreferrer noopener\">Doubly Robust Estimation in Missing Data and Causal Inference Models<\/a>,\u201d Biometrics<\/em>, 61 (4), 962\u201373.<\/p>\n\n\n\nBronnenberg, Bart J., Jean-Pierre Dub\u00b4e, and Robert E. Sanders (2020), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1287//mksc.2019.1189/" target=\"_blank\">Consumer misinformation and the brand premium: a private label blind taste test<\/a>,\u201d Marketing Science<\/em>, 39 (2), <a id=\"_bookmark22\"><\/a>382\u2013406.<\/p>\n\n\n\nCallaway, Brantly and Pedro H.C. Sant\u2019Anna (2021), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1016//j.jeconom.2020.12.001/" target=\"_blank\">Difference-in-differences with Multiple Time Periods<\/a>,\u201d Journal of Econometrics<\/em>, 225 (2), 200\u2013230.<a id=\"_bookmark23\"><\/a><\/p>\n\n\n\nCao, Jingcun, Pradeep Chintagunta, and Shibo Li (2023). \u201c<a href=https://www.ama.org/"https:////doi.org//10.1177//00222437221131562/" target=\"_blank\">From free to Paid: Monetizing a Non-Advertising-Based App<\/a>,\u201d Journal of Marketing Research<\/em>, 60 (4), 707\u201327.<a id=\"_bookmark24\"><\/a><\/p>\n\n\n\nCengiz, Doruk, Arindrajit Dube, Attila Lindner, and Ben Zipperer (2019), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1093//qje//qjz014/" target=\"_blank\" rel=\"noreferrer noopener\">The Effect of Minimum Wages on Low-Wage Jobs<\/a>,\u201d Quarterly Journal of Economics<\/em>, 134 (3), 1405\u201354.<\/p>\n\n\n\nChan, Marc K. and Simon C.M. Kwok (2016). \u201c<a href=https://www.ama.org/"https:////ideas.repec.org//p//syd//wpaper//2016-11.html/">Policy Evaluation with Interactive Fixed Effects<\/a>,\u201d working paper, University of Sydney.<\/p>\n\n\n\nChernozhukov, Victor, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins (2018), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1111//ectj.12097/" target=\"_blank\" rel=\"noreferrer noopener\">Double\/Debiased Machine Learning for Treatment and Structural Parameters<\/a>,\u201d Econometrics Journal <\/em>, 21 (1), C1\u2013C68.<\/p>\n\n\n\n<a id=\"_bookmark27\"><\/a>Cinelli, Carlos and Chad Hazlett (2020), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1111//rssb.12348/" target=\"_blank\">Making Sense of Sensitivity: Extending Omitted Variable Bias<\/a>,\u201d Journal of the Royal Statistical Society: Series B (Statistical Methodology)<\/em>, 82 (1), 39\u201367.<\/p>\n\n\n\nClarke, Damian, Daniel Pailanir, Susan Athey, and Guido Imbens (forthcoming), \u201cOn Synthetic Difference-in-Differences and Related Estimation Methods in Stata,\u201d Stata Journal.<\/em> <a id=\"_bookmark29\"><\/a><\/p>\n\n\n\nDanaher, Peter J. and Michael S. Smith (2010). \u201c<a href=https://www.ama.org/"https:////doi.org//10.1287//mksc.1090.0491/" target=\"_blank\">Modeling Multivariate Distributions Using Copulas: Applications in Marketing<\/a>,\u201d Marketing Science<\/em>, 30 (1), 4\u201321.<a id=\"_bookmark30\"><\/a><\/p>\n\n\n\nDiamond, Alexis and Jasjeet Sekhon (2013). \u201c<a href=https://www.ama.org/"https:////doi.org//10.1162//REST_a_00318/" target=\"_blank\">Genetic Matching for Estimating Causal Effects: A General Multivariate Matching Method for Achieving Balance in Observational Studies<\/a>,\u201d Review of Economics and Statistics<\/em>, 95 <\/em>(3), 932\u201345.<\/p>\n\n\n\nEckles, Dean and Eytan Bakshy (2021), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1080//01621459.2020.1796393/" target=\"_blank\" rel=\"noreferrer noopener\">Bias and High-Dimensional Adjustment in Observational Studies of Peer Effects<\/a>,\u201d Journal of the American Statistical Association<\/em>, 116 (534), 507\u201317.<\/p>\n\n\n\nEllickson, Paul B., Wreetabrata Kar, and James C. Reeder III (2023), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1287//mksc.2022.1401/" target=\"_blank\">Estimating Marketing Component Effects: Double Machine Learning from Targeted Digital Promotions<\/a>,\u201d Marketing Sci<a id=\"_bookmark33\"><\/a>ence<\/em>, 42 (4), 704\u201328.<\/p>\n\n\n\nFeder, Amir, Katherine A. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, et al. (2022). \u201c<a href=https://www.ama.org/"https:////doi.org//10.1162//tacl_a_00511/" target=\"_blank\">Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond<\/a>,\u201d Transactions of the Association for Computational Linguistics<\/em>, 10, 1138\u201358.<\/p>\n\n\n\nFong, Christian, Chad Hazlett, and Kosuke Imai (2018). \u201c<a href=https://www.ama.org/"https:////projecteuclid.org//journals//annals-of-applied-statistics//volume-12//issue-1//Covariate-balancing-propensity-score-for-a-continuous-treatment--Application//10.1214//17-AOAS1101.full/" target=\"_blank\">Covariate Balancing Propensity Score for a Continuous Treatment: Application to the Efficacy of Political Advertisements<\/a>,\u201d Annals of <a id=\"_bookmark35\"><\/a>Applied Statistics<\/em>, 12 (1), 156\u201377.<\/p>\n\n\n\nGhose, Anindya, Heeseung Andrew Lee, Kihwan Nam, and Wonseok Oh (2024), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1177//00222437231180494/" target=\"_blank\">The Effects of Pressure and Self-Assurance Nudges on Product Purchases and Returns in Online Retailing: Evidence from a Randomized Field Experiment<\/a>,\u201d Journal of Marketing Research<\/em>, 61 (3), 517\u201335.<\/p>\n\n\n\nGobillon, Laurent and Thierry Magnac (2016), \u201c<a href=https://www.ama.org/"https:////www.jstor.org//stable//24917033/" target=\"_blank\">Regional Policy Evaluation: Interactive Fixed Effects and Synthetic Controls<\/a>,\u201d Review of Economics and Statistics<\/em>, 98 (3), 535\u201351.<a id=\"_bookmark37\"><\/a><\/p>\n\n\n\nGoodman-Bacon, Andrew (2021), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1016//j.jeconom.2021.03.014/" target=\"_blank\">Difference-in-Differences with Variation in Treatment Timing<\/a>,\u201d <a id=\"_bookmark38\"><\/a>Journal of Econometrics<\/em>, 225 (2), 254\u201377.<\/p>\n\n\n\nGordon, Brett R., Florian Zettelmeyer, Neha Bhargava, and Dan Chapsky (2019), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1287//mksc.2018.1135/" target=\"_blank\">A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook<\/a>,\u201d Marketing Science<\/em>, 38 (2), 193\u2013225.<\/p>\n\n\n\nGormley, Todd A. and David A. Matsa (2011), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1093//rfs//hhr011/" target=\"_blank\">Growing Out of Trouble? Corporate Responses to Liability Risk<\/a>,\u201d Review of Financial Studies<\/em>, 24 (8), 2781\u20132821.<\/p>\n\n\n\nGuo, Tong, Srinivasaraghavan Sriram, and Puneet Manchanda (2020), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1287//mksc.2019.1181/" target=\"_blank\">\u2018Let the Sunshine In\u2019: The Impact of Industry Payment Disclosure on Physician Prescription Behavior<\/a>,\u201d Marketing Science<\/em>, <a id=\"_bookmark41\"><\/a>39 (3), 516\u201339.<\/p>\n\n\n\nGuo, Tong, Srinivasaraghavan Sriram, and Puneet Manchanda (2021), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1177//0022243720972106/" target=\"_blank\">The Effect of Information Disclosure on Industry Payments to Physicians<\/a>,\u201d Journal of Marketing Research<\/em>, 58 (1), 115\u201340.<\/p>\n\n\n\nHartford, Jason, Greg Lewis, Kevin Leyton-Brown, and Matt Taddy (2017), \u201c<a href=https://www.ama.org/"https:////proceedings.mlr.press//v70//hartford17a//hartford17a.pdf/" target=\"_blank\">Deep IV: A Flexible Approach for Counterfactual Prediction<\/a>,\u201d in Proceedings of the 34th International Conference on Machine Learning<\/em>, Vol. 70, D. Precup and Y.W. The, eds. <a id=\"_bookmark43\"><\/a>PMLR, 1414\u201323.<\/p>\n\n\n\nHeckman, James J., Hidehiko Ichimura, and Petra E. Todd (1997), \u201c<a href=https://www.ama.org/"https:////doi.org//10.2307//2971733/" target=\"_blank\">Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme<\/a>,\" <a id=\"_bookmark44\"><\/a>Review of Economic Studies<\/em>, 64 (4), 605\u201354.<\/p>\n\n\n\nHsiao, Cheng, H. Steve Ching, and Shui Ki Wan (2012), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1002//jae.1230/" target=\"_blank\">A Panel Data Approach for Program Evaluation: Measuring the Benefits of Political and Economic Integration of Hong Kong with Mainland China<\/a>,\u201d Journal of Applied Econometrics<\/em>, 27 (5), 705\u201340.<\/p>\n\n\n\nImbens, Guido W. (2003), \u201c<a href=https://www.ama.org/"https:////www.aeaweb.org//articles?id=10.1257\/000282803321946921\%22 target=\"_blank\">Sensitivity to Exogeneity Assumptions in Program Evaluation<\/a>,\u201d American Economic Review<\/em>, 93 (2), 126\u201332.<\/p>\n\n\n\nKeller, Kristopher O., Jonne Y. Guyt, and Rajdeep Grewal (2024), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1177//00222437231195551/" target=\"_blank\">Soda Taxes and Marketing Conduct<\/a>,\" <a id=\"_bookmark47\"><\/a>Journal of Marketing Research<\/em>, 61 (3), 393\u2013410.<\/p>\n\n\n\nKim, Sungjin, Clarence Lee, and Sachin Gupta (2020), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1177//0022243720936230/" target=\"_blank\">Bayesian Synthetic Control Methods<\/a>,\u201d Journal of Marketing Research<\/em>, 57 (5), 831\u201352.<\/p>\n\n\n\n<a id=\"_bookmark49\"><\/a>Lambrecht, Anja, Catherine Tucker, and Xu Zhang (2024), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1177//00222437231180171/" target=\"_blank\">TV Advertising and Online Sales: A Case Study of Intertemporal Substitution Effects for an Online Travel Platform<\/a>,\u201d Journal of <a id=\"_bookmark50\"><\/a>Marketing Research<\/em>, 61 (2), 248\u201370.<\/p>\n\n\n\nLi, Kathleen T. (2020), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1080//01621459.2019.1686986/" target=\"_blank\">Statistical Inference for Average Treatment Effects Estimated by Synthetic Control Methods<\/a>,\u201d Journal of the American Statistical Association<\/em>, 115 (532), 2068\u2013<a id=\"_bookmark51\"><\/a>83.<\/p>\n\n\n\nLi, Kathleen T. (2024), \"<a href=https://www.ama.org/"https:////doi.org//10.1287//mksc.2022.0212/" target=\"_blank\">Frontiers: A Simple Forward Difference-in-Differences Method<\/a>,\u201d Marketing <\/em><a id=\"_bookmark52\"><\/a>Science<\/em>, 43 (2), 267\u201379.<\/p>\n\n\n\nLi, Kathleen T. and Garrett P. Sonnier (2023), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1177//00222437221137533/" target=\"_blank\">Statistical Inference for Factor Model Approach to Estimate Causal Effects in Quasi-Experimental Settings<\/a>,\u201d Journal of Marketing Research<\/em>, 60 (3), <a id=\"_bookmark53\"><\/a>449\u201372.<\/p>\n\n\n\nLi, Kathleen T. and Christophe Van den Bulte (2023), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1287//mksc.2022.1406/" target=\"_blank\">Augmented Difference-in-Differences<\/a>,\u201d Marketing <\/em><a id=\"_bookmark54\"><\/a>Science<\/em>, 42 (4), 746\u201367.<\/p>\n\n\n\nLi, Sheng, Nikos Vlassis, Jaya Kawale, and Yun Fu (2016), \u201c<a href=https://www.ama.org/"https:////dl.acm.org//doi//abs//10.5555//3061053.3061146/" target=\"_blank\">Matching via Dimensionality Reduction for Estimation of Treatment Effects in Digital Marketing Campaigns<\/a>,\u201d in Proceedings of the <a id=\"_bookmark55\"><\/a>25th International Joint Conference on Artificial Intelligence (IJCAI)<\/em>. IJCAI, 3768\u201374.<\/p>\n\n\n\nLiu, Weiwei, S. Janet Kuramoto, and Elizabeth A. Stuart (2013), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1007//s11121-012-0339-5/" target=\"_blank\">An Introduction to Sensitivity Analysis for Unobserved Confounding in Nonexperimental Prevention Research<\/a>,\u201d Prevention Science<\/em>, 14, 570\u201380.<\/p>\n\n\n\nLovett, Mitchell J., Renana Peres, and Linli Xu (2019), \u201c<a href=https://www.ama.org/"https:////link.springer.com//article//10.1007//s11129-019-09211-9/" target=\"_blank\">Can Your Advertising Really Buy Earned Impressions? The Effect of Brand Advertising on Word of Mouth<\/a>,\u201d Quantitative Marketing <a id=\"_bookmark57\"><\/a>and Economics<\/em>, 17, 215\u201355.<\/p>\n\n\n\nMasten, Matthew A. and Alexandre Poirier (2022), \u201c<a href=https://www.ama.org/"https:////doi.org//10.48550//arXiv.2208.00552/" target=\"_blank\" rel=\"noreferrer noopener\">The Effect of Omitted Variables on the Sign of Regression Coefficients<\/a>,\u201d arXiv<\/em>, <a href=https://www.ama.org/"https:////doi.org//10.48550//arXiv.2208.00552/" target=\"_blank\">https:\/\/doi.org\/10.48550\/arXiv.2208.00552<\/a>.<\/p>\n\n\n\nNickerson, Dionne, Michael Lowe, Adithya Pattabhiramaiah, and Alina Sorescu (2023), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1177//00222429211044155/" target=\"_blank\">The Impact of Corporate Social Responsibility on Brand Sales: An Accountability Perspective<\/a>,\u201d Journal <a id=\"_bookmark59\"><\/a>of Marketing<\/em>, 87 (1), 5\u201328.<\/p>\n\n\n\nOster, Emily (2019), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1080//07350015.2016.1227711/" target=\"_blank\">Unobservable Selection and Coefficient Stability: Theory and Evidence<\/a>,\u201d Journal of Business Economic Statistics<\/em>, 37 (2), 187\u2013204.<a id=\"_bookmark60\"><\/a><\/p>\n\n\n\nPang, Xun, Licheng Liu, and Yiqing Xu (2022), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1017//pan.2021.22/" target=\"_blank\">A Bayesian Alternative to Synthetic Control for Comparative Case Studies<\/a>,\u201d Political Analysis<\/em>, 30 (2), 269\u201388.<\/p>\n\n\n\nPark, Sungho and Sachin Gupta (2012), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1287//mksc.1120.0718/" target=\"_blank\">Handling Endogenous Regressors by Joint Estimation Using Copulas<\/a>,\u201d Marketing Science<\/em>, 31 (4), 567\u201386.<\/p>\n\n\n\nPattabhiramaiah, Adithya, S. Sriram, and Puneet Manchanda (2019), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1177//0022242918815163/" target=\"_blank\" rel=\"noreferrer noopener\">Paywalls: Monetizing Online Content<\/a>,\u201d Journal of Marketing<\/em>, 83 (2), 19\u201336.<\/p>\n\n\n\nPattabhiramaiah, Adithya., Eric Overby, and Lizhen Xu (2022), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1287//mnsc.2021.4092/" target=\"_blank\">Spillovers from Online Engagement: How a Newspaper Subscriber\u2019s Activation of Digital Paywall Access Affects Her Retention and Subscription Revenue<\/a>,\u201d Management Science<\/em>, 68 (5), 3528\u201348.<a id=\"_bookmark64\"><\/a><\/p>\n\n\n\nPetrin, Amil and Kenneth Train (2010), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1509//jmkr.47.1.3/" target=\"_blank\">A Control Function Approach to Endogeneity in Consumer Choice Models<\/a>,\u201d Journal of Marketing Research<\/em>, 47 (1), 3\u201313.<a id=\"_bookmark65\"><\/a><\/p>\n\n\n\nRamachandra, Vikas (2018), \u201c<a href=https://www.ama.org/"https:////doi.org//10.48550//arXiv.1803.00149/" target=\"_blank\">Deep Learning for Causal Inference<\/a>,\u201d arXiv.org, https:\/\/doi.org\/10.48550\/arXiv.1803.00149.<\/p>\n\n\n\nRobins, James, Andrea Rotnitzky, and Lue Ping Zhao (1994), \u201c<a href=https://www.ama.org/"https:////doi.org//10.2307//2290910/" target=\"_blank\">Estimation of Regression Coefficients When Some Regressors Are Not Always Observed<\/a>,\u201d Journal of the American Statistical Association<\/em>, 89 (427), 846\u201366.<\/p>\n\n\n\nRosenbaum, P.R. and D.B. Rubin (1983), \u201c<a href=https://www.ama.org/"https:////www.jstor.org//stable//2345524/" target=\"_blank\">Assessing Sensitivity to an Unobserved Binary Covariate in an Observational Study with Binary Outcome<\/a>,\u201d Journal of the Royal <a id=\"_bookmark68\"><\/a>Statistical Society: Series B (Methodological)<\/em>, 45 (2), 212\u201318.<\/p>\n\n\n\nRoth, Jonathan, Pedro H.C. Sant\u2019Anna, Alyssa Bilinski, and John Poe (2023), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1016//j.jeconom.2023.03.008/" target=\"_blank\">What\u2019s Trending in Difference-in-Differences? A Synthesis of the Recent Econometrics Literature<\/a>,\u201d Journal <a id=\"_bookmark69\"><\/a>of Econometrics<\/em>, 233 (2), 345\u201366.<\/p>\n\n\n\nSant\u2019Anna, Pedro H.C. and Jun Zhao (2020), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1016//j.jeconom.2020.06.003/" target=\"_blank\">Doubly Robust Difference-in-Differences Estimators<\/a>,\u201d<a id=\"_bookmark70\"><\/a> Journal of Econometrics<\/em>, 219 (1), 101\u201322.<\/p>\n\n\n\nSingh, Amandee, Kartik Hosanagar, and Amit Gandhi (2020), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1145//3391403.3399466/" target=\"_blank\">Machine Learning Instrument Variables for Causal Inference<\/a>,\u201d in Proceedings of the 21st ACM Conference on Economics and <a id=\"_bookmark71\"><\/a>Computation (EC \u201920)<\/em>. ACM, <\/em>400\u2013417<\/p>\n\n\n\nSun, Liyang and Sarah Abraham (2021). \u201c<a href=https://www.ama.org/"https:////doi.org//10.1016//j.jeconom.2020.09.006/" target=\"_blank\">Estimating Dynamic Treatment Effects in Event Studies with Heterogeneous Treatment Effects<\/a>,\u201d Journal of Econometrics<\/em>, 225 (2), 175\u201399.<\/p>\n\n\n\nWager, Stefan and Susan Athey (2018), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1080//01621459.2017.1319839/" target=\"_blank\">Estimation and Inference of Heterogeneous Treatment Effects Using Random Forests<\/a>,\u201d Journal of the American Statistical Association<\/em>, 113 (523), <a id=\"_bookmark73\"><\/a>1228\u201342.<\/p>\n\n\n\nWooldridge, Jeffrey M. (2019). Introductory Econometrics: A Modern Approach<\/em>, <\/em>7th ed.<a><\/a> Cengage Learning.<\/p>\n\n\n\nXu, Yiqing (2017), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1017//pan.2016.2/" target=\"_blank\">Generalized Synthetic Control Method: Causal Inference with Interactive Fixed Effects Models<\/a>,\u201d Political Analysis<\/em>, 25 (1), 57\u201376.<\/p>\n\n\n\nYao, Liuyi, Sheng Li, Yaliang Li, Mengdi Huai, Jing Gao, and Aidong Zhang (2018), \u201c<a href=https://www.ama.org/"https:////dl.acm.org//doi//10.5555//3327144.3327188/" target=\"_blank\">Representation Learning for treatment effect estimation from observational data<\/a>,\u201d in Advances in <a id=\"_bookmark76\"><\/a>Neural Information Processing Systems<\/em>. NeurIPS, 2633\u201343.<\/p>\n\n\n\nZhang, Mengxia and Lan Luo (2023), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1287//mnsc.2022.4359/" target=\"_blank\">Can Consumer-Posted Photos Serve as a Leading Indicator of Restaurant Survival? Evidence from Yelp<\/a>,\u201d Management Science<\/em>, 69 (1), 25\u201350.<\/p>\n\n\n\nZubizarreta, Jos\u00e9 R. (2015), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1080//01621459.2015.1023805/" target=\"_blank\">Stable Weights That Balance Covariates for Estimation with Incomplete Outcome Data<\/a>,\u201d Journal of the American Statistical Association<\/em>, 110 (511), 910\u201322.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n[1]<\/sup>Researchers looking to overcome issues related to heterogeneous treatment effects in staggered treatment settings and interpret the composite treatment effect across all cohorts might also consider the synthetic control method that we discuss in the next section. Synthetic control methods do not suffer from this problem, as they estimate a separate ATT for each treatment group and use \u201cclean\u201d controls (units that have never been treated) for computing the ATT for the entire sample.<\/p>\n\n\n\nGo to the <a href=https://www.ama.org/"https:////www.ama.org//journal-of-marketing-research///" target=\"_blank\">Journal of Marketing Research<\/a><\/em><\/p>\n\n\n\n<div style=\"height:40px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n\n\n<div style=\"height:40px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n","post_title":"Causal Inference with Quasi-Experimental Data","post_excerpt":"This article provides an overview of the methodological toolkit available to empirical researchers who are interested in making causal inference using quasi-experimental data.","post_status":"publish","comment_status":"closed","ping_status":"closed","post_password":"","post_name":"causal-inference-with-quasi-experimental-data","to_ping":"","pinged":"","post_modified":"2024-11-25 13:43:57","post_modified_gmt":"2024-11-25 19:43:57","post_content_filtered":"","post_parent":0,"guid":"https:\/\/www.ama.org\/?post_type=ama_marketing_news&p=175179","menu_order":0,"post_type":"ama_marketing_news","post_mime_type":"","comment_count":"0","filter":"raw"},{"ID":171108,"post_author":"193730","post_date":"2024-09-30 10:44:38","post_date_gmt":"2024-09-30 15:44:38","post_content":"\nResearchers often wish to complement observed choice data with survey information to gain a deeper understanding of the underlying attitudes, motivations, and mechanisms that drive consumption behavior. Surveys are fundamental to obtain information that is unavailable through objectively recorded official statistics or about behaviors that are hard to measure. However, consumers are increasingly reluctant to share information about themselves in surveys: frequent privacy violations and breaches have made them more circumspect. Furthermore, topics that were once nonsensitive have now become stigmatized or politicized. For instance, consumers might claim to make sustainable or \u201cgreen\u201d consumption choices due to social desirability (<a href=https://www.ama.org/"https:////doi.org//10.1177//0022242919825649/" target=\"_blank\" rel=\"noreferrer noopener\">White, Habib, and Hardisty 2019<\/a>). Due to these trends, obtaining accurate information about sensitive topics has become increasingly challenging to analysts.<\/p>\n\n\n\n\n\nThis report summarizes our framework of strategies that are available to analysts to enhance truthful and accurate survey responses on sensitive topics. We present a decision tree to aid survey design for sensitive topics. The decision tree addresses presurvey administration, question design to improve truth-telling (survey planning), and statistical corrections after data collection (postsurvey adjustments).<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img src=https://www.ama.org/"https:////www.ama.org//wp-content//uploads//2024//09//gregori-figure-2.png/" alt=\"\" class=\"wp-image-171134\"\/><\/figure>\n\n\n\nWe complement the decision tree with eight survey techniques to reduce social desirability bias, summarized in the table below. We rate each of these techniques on the following criteria that influence truthful reporting:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Privacy protection<\/em>:<\/strong> The degree of confidentiality guaranteed to survey participants, where \u201cvery high\u201d indicates that the survey designer cannot infer the individual answer.<\/li>\n\n\n\n<li>Efficiency<\/em>:<\/strong> Some techniques introduce noise in the response to hide participants\u2019 answers. As a consequence, accuracy will be lower and statistics of interest will be imperfectly estimated, with large standard errors.<\/li>\n\n\n\n<li>Affective costs<\/em>:<\/strong> Some techniques might require survey participants to (seemingly) incriminate themselves which may be emotionally stressful.<\/li>\n\n\n\n<li>Cognitive costs<\/em>:<\/strong> Instructions of some techniques may be difficult for survey participants to understand.<\/li>\n\n\n\n<li>Design complexity (behavioral costs)<\/em>:<\/strong> Some techniques may be complex to implement, requiring much effort for appropriate implementation.<\/li>\n<\/ul>\n\n\n\nHigh affective and cognitive costs may result in systematic bias and\/or survey attrition: uncomfortable or confused participants may drop out, or provide untruthful or random answers. The tree requires analysts to consider the alphabetically labeled questions as well as their implications for the survey design and analysis; in addition, it introduces several techniques to reduce social desirability bias. We elaborate on each numbered step in the tree.<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Use direct questions if the topic is not sensitive<\/em>.<\/strong> Before introducing unnecessary complexity in the survey design, topic sensitivity should be assessed (<a href=https://www.ama.org/"https:////doi.org//10.1037//0033-2909.133.5.859/" target=\"_blank\" rel=\"noreferrer noopener\">Tourangeau and Yan 2007<\/a>). Perhaps what the analysts consider to be a sensitive topic is not as sensitive for survey participants (for instance, soft drug use among students). If in doubt, a pretest can be used to ask a sample of eligible participants to evaluate the sensitivity of certain topics or questions. Alternatively, the survey can ask participants to evaluate topic sensitivity ex post and assess whether the reported sensitivity is correlated with the responses to sensitive questions. If evidence of topic sensitivity is found, this can be reduced by building trust with the participant. <\/li>\n\n\n\n<li>Stress anonymity and allow for opt-out<\/em>.<\/strong> Building trust with the survey participant before the actual survey is essential to obtain accurate responses about potentially sensitive topics. Trust can be enhanced by using easily understandable confidentiality and ethics guarantees and by clearly stating the purpose of the survey and the intended data use (<a href=https://www.ama.org/"https:////doi.org//10.1086//269458/" target=\"_blank\" rel=\"noreferrer noopener\">Singer, Thurn, and Miller 1995<\/a>; <a href=https://www.ama.org/"https:////pubmed.ncbi.nlm.nih.gov//9243569///">Tourangeau et al. 1997<\/a>; <a href=https://www.ama.org/"https:////doi.org//10.1037//0033-2909.133.5.859/" target=\"_blank\" rel=\"noreferrer noopener\">Tourangeau and Yan 2007<\/a>). These guarantees should emphasize participants\u2019 legal right to privacy and state that participants may opt out from participating at any time during the survey. Further, these guarantees should stress the importance of participants\u2019 answers to better understand behavioral phenomena. <a href=https://www.ama.org/"https:////doi.org//10.1146//annurev-economics-091622-010157/" target=\"_blank\" rel=\"noreferrer noopener\">Stantcheva (2023)<\/a> provides more guidelines and suggestions. Data quality can also be enhanced by asking participants to commit to provide thoughtful answers (<a href=https://www.ama.org/"https:////www.qualtrics.com//blog//attention-checks-and-data-quality///">Geisen 2022<\/a>), similarly to \u201chonesty pledges.\u201d Finally, trust can be built by allowing survey participants to bypass potentially sensitive questions. Skipping could be introduced with a \u201cprefer not to say\u201d answer option, or by allowing survey participants to skip a sensitive section entirely (<a>Pieters and De Jong 2024<\/a>). <\/li>\n\n\n\n<li>Reduce sensitivity of direct questions<\/em>.<\/strong> If indirect questions are not possible or optimal to obtain the information of interest, social desirability bias in direct questions can be reduced, for instance by using more neutral and less emotional words, or by extending the time frame of interest to a longer period. Open-ended questions allow participants more flexibility in terms of how much to disclose. <\/li>\n\n\n\n<li>Use indirect evaluation questions<\/em>:<\/strong> Consumers are sometimes not consciously aware of their true behaviors or preferences\u2014for instance, when reporting on snacking frequency, stereotyping or gender issues. At the same time, they project their own behaviors, beliefs, and evaluations onto others (<a href=https://www.ama.org/"https:////doi.org//10.1086//209351/" target=\"_blank\" rel=\"noreferrer noopener\">Fisher 1993<\/a>). Indirect evaluation questions exploit their inability to gauge their true preferences and evaluation biases to obtain truthful responses to sensitive questions. For instance, \u201csocial circle\u201d questions ask about the behavior of neighbors or friends. Social circle questions improved survey polling for the U.S. presidential election in 2016 (<a href=https://www.ama.org/"https:////www.nature.com//articles//s41562-018-0302-y/" target=\"_blank\" rel=\"noreferrer noopener\">Galesic et al. 2018<\/a>). Related to this approach, the Bayesian truth serum asks survey participants to predict their own behavior as well as its prevalence in the group. Survey participants are compensated according to the accuracy of their prediction, creating incentives for accurate reporting (<a href=https://www.ama.org/"https:////doi.org//10.1509//jmr.09.0039/" target=\"_blank\" rel=\"noreferrer noopener\">Weaver and Prelec 2013<\/a>). A related indirect question approach is the endorsement experiment, where survey participants are asked to endorse a person or organization. These are randomly associated (or not) with a controversial policy or topic (e.g., abortion in <a href=https://www.ama.org/"https:////doi.org//10.1111//ajps.12205/" target=\"_blank\" rel=\"noreferrer noopener\">Rosenfeld, Imai, and Shapiro [2016]<\/a>). Endorsement of a certain organization or person may thus reveal hidden information about participants\u2019 thoughts on certain sensitive topics. <\/li>\n\n\n\n<li>Mask responses for sample-level information (simple)<\/em>.<\/strong> If aggregate data at the sample level are sufficient, responses can be simply masked, so that the individual response becomes unobservable. For instance, list experiments ask survey participants to answer how many questions from a set of questions they endorse (De Jong and Pieters 2019). Participants are randomly assigned to two conditions, each with the same baseline k<\/em> questions, but with an extra-sensitive question in one of the conditions. The analyst can then simply infer from the difference in average response between conditions which proportion of the sample endorses the sensitive question. Randomized response techniques hide answers by using randomization devices, such as (electronic) dice or coins, whose outcome is masked to the survey designer (<a href=https://www.ama.org/"https:////doi.org//10.1509//jmkr.47.1.14/">De Jong, Pieters, and Fox 2010<\/a>; <a href=https://www.ama.org/"https:////doi.org//10.1287//isre.1090.0271/" target=\"_blank\" rel=\"noreferrer noopener\">Kwan, So, and Tam 2010<\/a>). For instance, they might randomly assign survey participants to a sensitive or a nonsensitive question, making unclear which question the participant is actually answering. The Crosswise Model presents instead survey participants with a pair of items, one sensitive and one nonsensitive, after which they are asked whether the answers to the two statements are the same or different (<a href=https://www.ama.org/"https:////link.springer.com//article//10.1007//s00184-007-0131-x/">Yu, Tian, and Tang 2008<\/a>). These techniques have lower efficiency, as they introduce additional noise, and thus are best suited when survey topics are strongly socially desirable, such as tobacco smoking among pregnant mothers. <\/li>\n\n\n\n<li>Mask responses for individual-level information (advanced)<\/em>.<\/strong> When individual-level data on sensitive topics is needed, and the previous steps of our framework have been taken, analysts could make use of new survey techniques to obtain individual-level data from response masking. These new techniques use statistical techniques to infer probabilistic individual-level data from masked survey responses, while still protecting participant privacy (De Jong and Pieters 2019; <a href=https://www.ama.org/"https:////doi.org//10.1177//00222437231205252/" target=\"_blank\" rel=\"noreferrer noopener\">Gregori, De Jong, and Pieters 2024<\/a>; <a href=https://www.ama.org/"https:////www.jstor.org//stable//24771848/" target=\"_blank\" rel=\"noreferrer noopener\">Kuha and Jackson 2014<\/a>). The key idea behind these techniques is that the probability that a participant provides specific answers to the baseline questions (in case of list surveys, but similar ideas hold for other techniques) can be estimated from other information in the survey, and from that the participant\u2019s response to the sensitive questions can be inferred probabilistically. <\/li>\n\n\n\n<li>Use direct questions with social desirability responding corrections<\/em>.<\/strong> The previous set of techniques aim to prevent<\/em> social desirability bias. Early literature on social desirability responding (SDR) introduced partial correlation techniques to correct biased responses to sensitive, direct questions. First, participants\u2019 inclination to provide socially acceptable responses, that is, their propensity to SDR, is assessed using so-called lie scales. These lie scales contain items such as \u201cI have not always been honest with myself\u201d or \u201cI sometimes tell lies if I have to\u201d (<a href=https://www.ama.org/"https:////doi.org//10.1177//2158244015621113/" target=\"_blank\" rel=\"noreferrer noopener\">Hart et al. 2015<\/a>). SDR propensity can thus be quantified and can then be included as a covariate when predicting the responses to sensitive questions. This should correct for variation in the outcome variable due to SDR propensity. Importantly, SDR correction of potentially biased responses to direct questions about sensitive topics is commonly not very effective and the extent of validity improvement often unknown (<a href=https://www.ama.org/"https:////doi.org//10.1111//jopy.12662/" target=\"_blank\" rel=\"noreferrer noopener\">Lanz, Thielmann, and Gerpott 2022<\/a>). <a href=https://www.ama.org/"https:////doi.org//10.1509//jmkr.47.2.199/" target=\"_blank\" rel=\"noreferrer noopener\">Steenkamp et al. (2010)<\/a> provide a procedure for using SDR scales to improve validity of marketing construct.<\/li>\n<\/ol>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-summary\">Summary<\/strong><\/h2>\n\n\n\nEnhancing perceived and actual privacy of survey participants is a rich, active research area. In this short article, we have outlined a number of strategies that can be utilized to build trust with survey participants and ultimately elicit truthful responses to sensitive questions. These techniques should be combined with ex post data processing to ensure that sensitive individual information cannot be deanonymized when making the data available (see, e.g., <a href=https://www.ama.org/"https:////www.ama.org//2022//03//17//the-marketer-at-the-privacy-table///" target=\"_blank\" rel=\"noreferrer noopener\">Gupta, Moutafis, and Schneider 2022<\/a>). Furthermore, additional issues need to be considered when designing surveys; in particular, respondents\u2019 fatigue, lack of engagement, or boredom might reduce the external validity of survey results (see <a href=https://www.ama.org/"https:////doi.org//10.1177//00222437211073581/" target=\"_blank\" rel=\"noreferrer noopener\">Li et al. 2022<\/a>). We hope this short introduction provides some accessible guidance towards obtaining sensitive information from consumers.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"h-references\">References<\/strong><\/h4>\n\n\n\nDe Jong, Martijn G., and Rik Pieters (2019), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1177//0022243718821312/" target=\"_blank\" rel=\"noreferrer noopener\">Assessing Sensitive Consumer Behavior Using the Item Count Response Technique<\/a>,\u201d Journal of Marketing Research<\/em>, 56 (3), 345\u201360. https:\/\/doi.org\/10.1177\/0022243718821312<\/p>\n\n\n\nDe Jong, Martijn G., Rik Pieters, and Jean-Paul Fox (2010), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1509//jmkr.47.1.14/" target=\"_blank\" rel=\"noreferrer noopener\">Reducing Social Desirability Bias Through Item Randomized Response: An Application to Measure Underreported Desires<\/a>,\u201d Journal of Marketing Research<\/em>, 47<\/em>(1), 14\u201327. https:\/\/doi.org\/10.1509\/jmkr.47.1.14<\/p>\n\n\n\nFisher, Robert J. (1993). \u201c<a href=https://www.ama.org/"https:////doi.org//10.1086//209351/" target=\"_blank\" rel=\"noreferrer noopener\">Social Desirability Bias and the Validity of Indirect Questioning<\/a>,\u201d Journal of Consumer Research<\/em>, 20 (2), 303\u201315. https:\/\/doi.org\/10.1086\/209351<\/p>\n\n\n\nGalesic, M., W. Bruine de Bruin, M. Dumas, A. Kapteyn, J.E. Darling, and E. Meijer (2018), \u201c<a href=https://www.ama.org/"https:////www.nature.com//articles//s41562-018-0302-y/" target=\"_blank\" rel=\"noreferrer noopener\">Asking About Social Circles Improves Election Predictions<\/a>,\u201d Nature Human Behaviour<\/em>, 2 (3), 187\u201393. https:\/\/www.nature.com\/articles\/s41562-018-0302-y<\/p>\n\n\n\nGeisen, Emily (2022), \u201c<a href=https://www.ama.org/"https:////www.qualtrics.com//blog//attention-checks-and-data-quality///" target=\"_blank\" rel=\"noreferrer noopener\">Improve Data Quality by Using Commitment Request Instead of Attention Checks<\/a>,\u201d Qualtrics XM Blog<\/em> (August 4), https:\/\/www.qualtrics.com\/blog\/attention-checks-and-data-quality\/.<\/p>\n\n\n\nGregori, Marco, Martijn G. De Jong, and Rik Pieters (2024), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1177//00222437231205252/" target=\"_blank\" rel=\"noreferrer noopener\">Response Aggregation to Obtain Truthful Answers to Sensitive Questions: Estimating the Prevalence of Illegal Purchases of Prescription Drugs<\/a>,\u201d Journal of Marketing Research<\/em>, https:\/\/doi.org\/10.1177\/00222437231205252<\/p>\n\n\n\nGupta, Sachin, Panos Moutafis, and Matthew J. Schneider (2022), \u201c<a href=https://www.ama.org/"https:////www.ama.org//2022//03//17//the-marketer-at-the-privacy-table///">The Marketer at the Privacy Table<\/a>,\u201d Impact at JMR<\/em> (March 17), https:\/\/www.ama.org\/2022\/03\/17\/the-marketer-at-the-privacy-table\/.<\/p>\n\n\n\nHart, Claire M., Timothy D. Ritchie, Erica G. Hepper, and Jochen E. Gebauer (2015), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1177//2158244015621113/" target=\"_blank\" rel=\"noreferrer noopener\">The Balanced Inventory of Desirable Responding Short Form (BIDR-16)<\/a>,\u201d Sage Open<\/em>, 5(4), https:\/\/doi.org\/10.1177\/2158244015621113.<\/p>\n\n\n\nKuha, Jouni and Jonathan Jackson (2014), \u201c<a href=https://www.ama.org/"https:////www.jstor.org//stable//24771848/" target=\"_blank\" rel=\"noreferrer noopener\">The Item Count Method for Sensitive Survey Questions: Modelling Criminal Behaviour<\/a>,\" Journal of the Royal Statistical Society Series C: Applied Statistics<\/em>, 63 (2), 321\u201341. https:\/\/www.jstor.org\/stable\/24771848<\/p>\n\n\n\nKwan, Samuel S., Mike K. So, and Kar Yan Tam (2010), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1287//isre.1090.0271/" target=\"_blank\" rel=\"noreferrer noopener\">Research note\u2014Applying the Randomized Response Technique to Elicit Truthful Responses to Sensitive Questions in IS Research: The Case of Software Piracy Behavior<\/a>,\" Information Systems Research<\/em>, 21 (4), 941\u201359. https:\/\/doi.org\/10.1287\/isre.1090.0271<\/p>\n\n\n\nYu, Jun-Wu, Guo-Liang Tian, and Man-Lai Tang (2008), \u201c<a href=https://www.ama.org/"https:////link.springer.com//article//10.1007//s00184-007-0131-x/" target=\"_blank\" rel=\"noreferrer noopener\">Two New Models for Survey Sampling with Sensitive Characteristic: Design and Analysis<\/a>,\u201d Metrika<\/em>, 67 (2008), 251\u201363. https:\/\/link.springer.com\/article\/10.1007\/s00184-007-0131-x<\/p>\n\n\n\nLanz, Lukas, Isabel Thielmann, and Fabiola H. Gerpott (2022), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1111//jopy.12662/" target=\"_blank\" rel=\"noreferrer noopener\">Are Social Desirability Scales Desirable? A Meta\u2010Analytic Test of the Validity of Social Desirability Scales in the Context of Prosocial Behavior<\/a>,\u201d Journal of Personality<\/em>, 90 (2), 203\u201321. https:\/\/doi.org\/10.1111\/jopy.12662<\/p>\n\n\n\nLi, Ye, Antonia Krefeld-Schwalb, Daniel G. Wall, Eric J. Johnson, Olivier Toubia, and Daniel M. Bartels (2022), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1177//00222437211073581/" target=\"_blank\" rel=\"noreferrer noopener\">The More You Ask, the Less You Get: When Additional Questions Hurt External Validity<\/a>,\u201d Journal of Marketing Research<\/em>, 59 (5), 963\u201382. https:\/\/doi.org\/10.1177\/00222437211073581<\/p>\n\n\n\nMeckel, Katherine and Bradley Shapiro (2021), \u201c<a href=https://www.ama.org/"https:////www.nber.org//papers//w29462/" target=\"_blank\" rel=\"noreferrer noopener\">Depression and Shopping Behavior<\/a>,\u201d Working Paper No. w29462, National Bureau of Economic Research, https:\/\/www.nber.org\/papers\/w29462.<\/p>\n\n\n\nRosenfeld, Bryn, Kosuke Imai, and Jacob N. Shapiro (2016), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1111//ajps.12205/" target=\"_blank\" rel=\"noreferrer noopener\">An Empirical Validation Study of Popular Survey Methodologies for Sensitive Questions<\/a>,\u201d American Journal of Political Science<\/em>, 60 (3), 783\u2013802. https:\/\/doi.org\/10.1111\/ajps.12205<\/p>\n\n\n\nPieters, Rik and Martijn G. De Jong (2024), \u201cThe Commodification of Intimacy: Desires for Escort Services,\u201d working paper.<\/p>\n\n\n\nSinger, Eleanor, Dawn R. Von Thurn, and Esther R. Miller (1995), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1086//269458/" target=\"_blank\" rel=\"noreferrer noopener\">Confidentiality Assurances and Response: A Quantitative Review of the Experimental Literature<\/a>,\u201d Public Opinion Quarterly<\/em>, 59 (1), 66\u201377. https:\/\/doi.org\/10.1086\/269458<\/p>\n\n\n\nStantcheva, Stefanie (2023), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1146//annurev-economics-091622-010157/" target=\"_blank\" rel=\"noreferrer noopener\">How to Run Surveys: A Guide to Creating Your Own Identifying Variation and Revealing the Invisible<\/a>,\u201d Annual Review of Economics<\/em>, 15, 205\u201334. https:\/\/doi.org\/10.1146\/annurev-economics-091622-010157<\/p>\n\n\n\nSteenkamp, Jan-Benedict E.M., Martijn G. De Jong, and Hans Baumgartner (2010), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1509//jmkr.47.2.199/" target=\"_blank\" rel=\"noreferrer noopener\">Socially Desirable Response Tendencies in Survey Research<\/a>,\u201d Journal of Marketing Research<\/em>, 47 (2), 199\u2013214. https:\/\/doi.org\/10.1509\/jmkr.47.2.199<\/p>\n\n\n\nTourangeau, R., J.B. Jobe, W.F. Pratt, and K. Rasinski (1997), \u201c<a href=https://www.ama.org/"https:////pubmed.ncbi.nlm.nih.gov//9243569///" target=\"_blank\" rel=\"noreferrer noopener\">Design and Results of the Women's Health Study<\/a>,\u201d NIDA Research Monograph<\/em>, 167, 344\u201365. https:\/\/pubmed.ncbi.nlm.nih.gov\/9243569\/<\/p>\n\n\n\nTourangeau, Roger, and Ting Yan (2007), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1037//0033-2909.133.5.859/" target=\"_blank\" rel=\"noreferrer noopener\">Sensitive Questions in Surveys<\/a>,\u201d Psychological Bulletin<\/em>, 133 (5), 859\u201383. https:\/\/doi.org\/10.1037\/0033-2909.133.5.859<\/p>\n\n\n\nWeaver, Ray and Drazen Prelec (2013), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1509//jmr.09.0039/" target=\"_blank\" rel=\"noreferrer noopener\">Creating Truth-Telling Incentives with the Bayesian Truth Serum<\/a>,\u201d Journal of Marketing Research<\/em>, 50 (3), 289\u2013302. https:\/\/doi.org\/10.1509\/jmr.09.0039<\/p>\n\n\n\nWetzel, Eunike, Susanne Frick, and Anna Brown (2021), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1037//pas0000971/" target=\"_blank\" rel=\"noreferrer noopener\">Does Multidimensional Forced-Choice Prevent Faking? Comparing The Susceptibility of the Multidimensional Forced-Choice Format and the Rating Scale Format to Faking<\/a>,\u201d Psychological Assessment<\/em>, 33 (2), 156\u201370. https:\/\/doi.org\/10.1037\/pas0000971<\/p>\n\n\n\nWhite, Katherine, Rishad Habib, and David J. Hardisty (2019), \u201c<a href=https://www.ama.org/"https:////doi.org//10.1177//0022242919825649/" target=\"_blank\" rel=\"noreferrer noopener\">How to SHIFT Consumer Behaviors to Be More Sustainable: A Literature Review and Guiding Framework<\/a>,\u201d Journal of Marketing<\/em>, 83 (3), 22\u201349. https:\/\/doi.org\/10.1177\/0022242919825649<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td style=\"line-height: 1;\">Decision Tree Step<\/strong><\/font><\/td> <td style=\"line-height: 1;\">Technique<\/strong><\/font><\/td> <td style=\"line-height: 1;\">Privacy Protection<\/strong><\/font><\/td> <td style=\"line-height: 1;\">Efficiency<\/strong><\/font><\/td> <td style=\"line-height: 1;\">Affective Costs<\/strong><\/font><\/td> <td style=\"line-height: 1;\">Cognitive Costs<\/strong><\/font><\/td> <td style=\"line-height: 1;\">Design Complexity<\/strong><\/font><\/td> <td style=\"line-height: 1;\">Example Formulation\/Reference<\/strong><\/font><\/td><\/tr><tr> <td style=\"line-height: 1;\">1. Direct questions<\/td> <td style=\"line-height: 1;\">i. Direct questions<\/td> <td style=\"line-height: 1;\">Very low<\/td> <td style=\"line-height: 1;\">Very high<\/td> <td style=\"line-height: 1;\">Very high<\/td> <td style=\"line-height: 1;\">Very low<\/td> <td style=\"line-height: 1;\">Very easy<\/td> <td style=\"line-height: 1;\"> <\/td><\/tr><tr> <td style=\"line-height: 1;\">2. Opt-out<\/td> <td style=\"line-height: 1;\">ii. Skip questions<\/td> <td style=\"line-height: 1;\">Low<\/td> <td style=\"line-height: 1;\">High<\/td> <td style=\"line-height: 1;\">Moderate<\/td> <td style=\"line-height: 1;\">Very low<\/td> <td style=\"line-height: 1;\">Very easy<\/td> <td style=\"line-height: 1;\">The next questions can be sensitive. It is no problem if you do not want to answer these questions. Please indicate here: \u201cproceed,\u201d \u201cskip these questions.\u201d See Pieters and De Jong (2024).<\/td><\/tr><tr><td style=\"line-height: 1;\" rowspan=\"2\">4. Indirect evaluation questions<\/td> <td style=\"line-height: 1;\">iii. Social circle questions<\/td> <td style=\"line-height: 1;\">Moderate<\/td> <td style=\"line-height: 1;\">High<\/td> <td style=\"line-height: 1;\">Moderate<\/td> <td style=\"line-height: 1;\">Low<\/td> <td style=\"line-height: 1;\">Very easy<\/td> <td style=\"line-height: 1;\">Indicate how many of your social contacts engage in the following behavior\/have the following attitudes.<\/td><\/tr><tr> <td style=\"line-height: 1;\">iv. Bayesian truth serum<\/td> <td style=\"line-height: 1;\">Moderate<\/td> <td style=\"line-height: 1;\">High<\/td> <td style=\"line-height: 1;\">Moderate<\/td> <td style=\"line-height: 1;\">Low<\/td> <td style=\"line-height: 1;\">Difficult<\/td> <td style=\"line-height: 1;\">How often do you <sensitive behavior>? Predict the percentage of respondents who choose each option. See Weaver and Prelec (2013).<\/td><\/tr><tr> <td style=\"line-height: 1;\"> <\/td> <td style=\"line-height: 1;\">v. Endorsement experiment<\/td> <td style=\"line-height: 1;\">Very high<\/td> <td style=\"line-height: 1;\">Very low<\/td> <td style=\"line-height: 1;\">Very low<\/td> <td style=\"line-height: 1;\">Low<\/td> <td style=\"line-height: 1;\">Difficult<\/td> <td style=\"line-height: 1;\">Do you have a favorable opinion of the following <organization\/person>? See Rosenfeld, Imai and Shapiro (2016).<\/td><\/tr><tr> <td style=\"line-height: 1;\">5\/6. Response masking<\/td> <td style=\"line-height: 1;\">vi. List experiment<\/td> <td style=\"line-height: 1;\">High<\/td> <td style=\"line-height: 1;\">Low<\/td> <td style=\"line-height: 1;\">Low<\/td> <td style=\"line-height: 1;\">Low<\/td> <td style=\"line-height: 1;\">Difficult<\/td> <td style=\"line-height: 1;\">How many of the next <number> statements are true?<\/td><\/tr><tr> <td style=\"line-height: 1;\"> <\/td> <td style=\"line-height: 1;\">vii. Randomized response<\/td> <td style=\"line-height: 1;\">Very high<\/td> <td style=\"line-height: 1;\">Low<\/td> <td style=\"line-height: 1;\">Moderate<\/td> <td style=\"line-height: 1;\">Moderate<\/td> <td style=\"line-height: 1;\">Moderate<\/td> <td style=\"line-height: 1;\">Toss a coin. Answer \u201cyes\u201d if the coin toss comes up heads. Otherwise, please answer the following question.<\/td><\/tr><tr> <td style=\"line-height: 1;\"> <\/td> <td style=\"line-height: 1;\">viii. Crosswise Model\/Paired Response Technique<\/td> <td style=\"line-height: 1;\">Very high<\/td> <td style=\"line-height: 1;\">Low<\/td> <td style=\"line-height: 1;\">Very low<\/td> <td style=\"line-height: 1;\">Low<\/td> <td style=\"line-height: 1;\">Moderate<\/td> <td style=\"line-height: 1;\">Are your answer to the next two statements the same or different? See Gregori, De Jong and Pieters (2023).<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\nNotes<\/em>: The table is based on Rosenfeld, Imai, and Shapiro (2016) and Gregori, De Jong, and Pieters (2023). Ranking of techniques is by the current authors based on overall superiority (most to least).<\/p>\n\n\n\nGo to the <a href=https://www.ama.org/"https:////www.ama.org//journal-of-marketing-research///" target=\"_blank\" rel=\"noreferrer noopener\">Journal of Marketing Research<\/a><\/em><\/p>\n\n\n\n<div style=\"height:40px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n\n\n<div style=\"height:40px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n","post_title":"Proven and New Survey Techniques to Obtain Sensitive Information from Consumers","post_excerpt":"Marketing professors Marco Gregori, Martijn G. de Jong, and Rik Pieters draw on a number of academic research studies to help analysts get more truthful and accurate survey responses on sensitive topics.","post_status":"publish","comment_status":"closed","ping_status":"closed","post_password":"","post_name":"proven-and-new-survey-techniques-to-obtain-sensitive-information-from-consumers","to_ping":"","pinged":"","post_modified":"2024-10-07 13:00:25","post_modified_gmt":"2024-10-07 18:00:25","post_content_filtered":"","post_parent":0,"guid":"https:\/\/www.ama.org\/?post_type=ama_marketing_news&p=171108","menu_order":0,"post_type":"ama_marketing_news","post_mime_type":"","comment_count":"0","filter":"raw"}]" />