Continuing with the ongoing blog series on causal inference with big data (part 1 & part 2 here), we pick where we last left off.
As a brief refresher, recall example 3. I explained the use of difference in differences (DD) methods is being applied to estimate the effect of management response to online reviews on subsequent opinion. Simply, by using two websites, one where managers respond to reviews and one where managers don’t, researchers can control underlying hotel quality using the latter website’s reviews to estimate the causal impact of the act of responding on subsequent ratings on the former website.
While this approach has tremendous intuitive appeal and comes from a rich tradition in econometric literature, we pointed out a potentially serious issue with the DD approach. The problem with the approach is that if managers endogenously time their response to random sequences of negative reviews, we would wrongly identify natural reversions to the mean in ratings as being caused by the manager’s response. In other words, the increase in observed ratings after managers respond could have occurred in the absence thereof.
As promised, I will describe how we resolve this selection endogeneity issue in my paper with Alex Chaudhry.
First, let’s think about how one would go about demonstrating the effect of management response on subsequent opinion in a lab setting. The following experiment is one likely possibility:
- Endow all subjects with the same experience.
- Make subjects rate the experience and write a review.
- In one group, show the subjects a management response to a prior review.
- In the second group, show the prior review but not the manager’s response.
- Compare the ratings between the 2 groups.
In the lab setting, the response is completely exogenous, i.e. randomly assigned. How can we do something similar using the raw data? In our paper, we propose the following natural experiment as the identification strategy:
- Find all reviews that immediately follow a previous review that received a response.
- Determine whether the response is written before or after the next review. This corresponds to whether the previous response is actually observable.
- Calculate the difference between the expected ratings of the reviews following an observable and an unobservable response.
Why do we say this approach accounts for the endogenous policy of response selection? Well, the two cases that we’re comparing both eventually get responses. So we are not comparing one group that gets responses to another group that doesn’t. We’re comparing one group that gets a response that is readily observed by the subsequent reviewer to another group that gets a response which isn’t readily observed by the subsequent reviewer.
In order to argue that the subsequent reviewer, who likely isn’t going on TripAdvisor to read reviews, is actually observing the management response. I offer the following screenshots as evidence that the responses are unavoidable if present.
Evidently, on both a typical desktop and mobile screen, one can generally see the next review and whether a management response is present just prior to clicking the button to write a review. This setting basically replicates a lab experiment. We are able to use this idiosyncratic website feature to test the effect of the mere existence of the management response on subsequent opinion.
What did we find? When the previous review was negative and the manager’s response was observed by the subsequent reviewer, we found that the subsequent rating was on average .13 stars higher. This is a economically significant amount considering that the majority of ratings are 4 stars. However, when the previous review was positive and the manager’s response was observed by the subsequent reviewer, we found that the subsequent rating was on average .03 stars lower.
How do we explain these results? Well, I think the easiest way is to give an analogous offline example. Imagine the following restaurant scenario:
You are dining during a relatively busy service period surrounded by occupied adjacent tables. Your waiter has done a fine job of serving your table in a very routine dining experience with no need to take exceptional actions. As you are finishing your meal and deliberating over the amount of tip to leave, you observe one of the following situations.
- In the first scenario, your waiter handles an adjacent table of fussy diners in a very professional and proactive manner.
- In the second scenario, your waiter just received a very handsome tip from an adjacent table and thanks the customers for their generous tip as they get up to leave.
How might your tip be influenced in each case?
If you’re anything like me, your prototypical representative economic agent, you might be swayed to leave a healthier tip to the first waiter who has shown himself to be excellent at his job through his publicly observable interactions with a different customer. In the second scenario, you might be swayed to leave a lower tip to the waiter who seems to be trying to influence everyone else’s tip. This second reaction is called reactance in the psychology literature. Reactance is the idea that when you sense your freedom to do something is being threatened by someone else, you will take actions to regain that freedom, often acting in a way counter to the intended direction of the individual attempting to influence your actions. In the example, the action is leaving a tip. In our study, the action is to write a review.
What do you think of our story? What alternative explanations would you come up with?
