Imagine a world where there is no trust. You have traveled a long way, perhaps on the Silk Road, and encounter a vendor that sells drinking water. Everyone traveling along this road is headed to a far-off land from which they will not return. You know there are 2 sources water this vendor could have used. One from the melting ice caps of a treacherous mountain and another from a nearby contaminated well. You cannot tell the difference between the two from sight, smell, or taste. However, drink the latter, and you are likely to contract a horrible disease. Under a non-life threatening need for water, would you buy from this vendor? If you’re rational and the vendor is rational, you should anticipate that the vendor has no incentive to collect the water from the more costly source (supposing that there is no cost of immorality). Therefore, there must be a market failure.
Now imagine there are 2 vendors at this location, one of whom operates as part of a brand that has locations all along the route. Which vendor would you buy from? If you and the vendors are rational, you will anticipate that the chain vendor has an incentive to sell the stated quality of water, since if you get sick before the next stop, you will not buy from the chain again and the chain loses a potential customer. The independent vendor still faces the same incentives, there’s nothing he can do to convince you that his water is from the mountain.
Enter “online” reviews. Suppose, as travelers journey forward, they send back verifiable messages via travelers going the opposite direction. These verified messages are posted on a board at each watering location for all to see. The messages contain information about the quality of the water purchased from each vendor. Now, what will you and the vendors do? Clearly, the independent vendor can also sell high quality water and verify it through the previous customers’ messages. This messaging system levels the playing field for the independent vendor and allows him to compete against the chain on quality.
In a recent research article, my coauthors (Alex & Amit) and I investigate the empirical validity of this logic applied to hotel markets. We formalize the argument with a stylized game theoretic model to show, mathematically, that equilibrium results can indeed lead to the situations described in the (anti)fairy tale example above. Then, we test this theory in the hotel market on a dataset of over 12 million reviews across 40k hotels in the US. Before I get into how we analyzed the data, let me preview our results. We find:
- On average, brands hold roughly a .16 star advantage in quality over similar independent hotels in the absence of online reviews.
- As TripAdvisor, the most prominent online review platform, reaches its peak penetration across markets, markets with median peak penetration levels saw increases in independent hotel quality of an average of .08 stars. This is economically meaningful as it can alter rank order of hotels on TripAdvisor’s site and also help push hotels to the next half star rounding point for aggregate ratings.
- Branded hotels do not improve their quality as a function of TripAdvisor penetration on average. This suggests that online reviews are helping independent hotels close the quality gap on their branded competitors.
- In subcategory ratings, we find that brands had the largest advantages in room quality, service, and cleanliness in the absence of online reviews. However, these are also the categories in which online reviews have caused independent hotels to gain the most ground.
- The location subcategory is one in which we should not expect differences in quality improvements due to the proliferation of online reviews. We find evidence of this null result, serving as a nice falsification test for our main results.
- Replacing TripAdvisor reviews with those from Expedia, where reviews are written by verified guests, we replicate our main results. This suggests that ratings improvements are not due to fake reviews, but actual quality improvements.
So, how did we get here? First, I begin with some of the major empirical challenges to this research.
Online reviews data is messy. We collected reviews from 42k domestic hotels on TripAdvisor. This yielded 12 million reviews. While this seems like a lot of data, per hotel, this is just an average of 300 reviews for each hotel. Spread this out over many years, the review data becomes somewhat sparse and noisy. Similarly, to measure market-level penetration of TripAdvisor, we use a measure that is roughly the ratio of reviews per available room at each point in time in each market. This measure can also be noisy. So our first challenge is to separate the information from the noise. We do this through state space modeling. In a nutshell, state space models is a method of analyzing time series data that deconstructs the data into a noisy measurement component driven by a predictable underlying latent component. The latent construct is modeled as the sum of several independent components. In our case, we modeled penetration and quality as a random walk + local trend with seasonality. This assumption of the underlying data generating process imposes a certain degree of “smoothness” and cyclicality on market penetration and hotel quality. I think these are good assumptions. We don’t expect market penetration or hotel quality to jump wildly, and certainly hotels experience seasonal trends (who wants to go to Houston in the summer!? Oh wait, me. But I digress). By separating measurement noise from the underlying latent construct, we are able to refine our measures of penetration and quality. We estimated the latent penetration and quality for thousands of markets and tens of thousands of hotels.
The figure above represents 4 market level penetration measures that we estimated. We superimposed the raw Google Trend index for TripAdvisor and its estimated latent trends for comparison. As you can see, the latent penetration trends are smooth while the raw Google index (as well as the raw penetration measure) are noisy and cyclical. Furthermore, we clearly have cross-sectional heterogeneity in penetration. Some markets, like Las Vegas, have much higher TripAdvisor usage than other markets, like Dallas. Moreover, there are rank order changes in latent penetration, e.g. Dallas surpasses Chicago by 2014. We will use this source of cross-sectional variation in penetration to study the impact of online review proliferation on hotel quality.
We estimated latent quality for over 40k hotels. Here’s an example of one that we picked at random. Clearly, the monthly average ratings (in blue) are quite noisy. Consecutive months can go from 4 stars to 2.5 stars. This cannot be true for actual quality! The latent quality smooths out these wild variations, being affected by only persistent changes in average ratings. There’s a graph like this for 40k hotels.
So now that we have addressed measurement issues, we approach the real problem. What if quality improvements are due to competition and not TripAdvisor and it just so happens that competition and TripAdvisor penetration move in the same direction over time? Additionally, what if competition, TripAdvisor usage, and grade inflation on TripAdvisor all occur simultaneously? This would mean that if there is a correlation between TripAdvisor penetration and hotel quality, it could be because of competition or grade inflation, and we’ve just wasted a lot of time.
In order to resolve this issue, we thought of the following thought experiment. What if there are 2 hotels that compete directly against one another, but somehow randomly got assigned to different TripAdvisor markets so that when one market experiences a lot of searches and reviews on TripAdvisor, only one of the hotels is exposed to this attention. But TripAdvisor markets are not randomly assigned. When you look up your next vacation resort in Vegas, you get a list of Vegas hotels. However, next to Las Vegas is the suburb of Henderson. It just so happens that Henderson also has a lot of hotels (and they get a lot of reviews relative to their size!). Henderson hotels do not show up on TripAdvisor searches for Vegas and vice versa. What if some hotel “markets” actually cross the municipal boundary? The “Boulder Strip” in Las Vegas is a prime example of this. This strip of hotel-casinos run right across the boundary of Henderson and Las Vegas. The Boulder Strip hotels on the Vegas side no more compete with strip hotels than Henderson hotels, though they will show up on a search for Vegas hotels and the ones in Henderson will not. If only there were a way to magically identify these natural geographic clusters of hotels.
Enter HDBSCAN. This is a magical clustering algorithm and I won’t go into too much detail here, but the algorithm takes in a minimum cluster size parameter and outputs cluster membership. Look at the example below. Here, we have generated some random sets of points that any human can easily cluster. There are 2 relatively dense clusters below which is a long curvy cluster and below which is a less dense cluster. There seems to be 2 outlier points. HDBSCAN (top row) easily identifies all 4 clusters and outliers while a common clustering algorithm, k-means, struggles to do so.
Applied to hotel GPS coordinates, we are able to identify clusters of hotel markets all around the US! Many of these clusters span several TripAdvisor markets. This allows us to refine our previous thought experiment.
Imagine hotel A and B are similar hotels competing in the same market identified by HDBSCAN. However, they fall under different TripAdvisor markets. If market 1 has a lot more exposure to TripAdvisor than market 2, then A should have more incentive to compete on quality than B. Moreover, if A and B are independent hotels and A’ and B’ are branded hotels, then the difference in quality between A and B should be larger than between A’ and B’ since brands did not need online reviews to signal their quality. We operationalize this thought experiment in a high-dimensional fixed effects regression where we control for fixed effects of market X time X hotel class as well as TripAdvisor market. This essentially allows us to compare hotels located in the same market, at the same time, of the same class (Four Seasons vs. Ritz Carlton), that just happen to belong to different TripAdvisor markets that have different levels of penetration. This regression allows us to document the findings listed above.
Now you know what we did and what we found, what’s the takeaway? I’ll start with the most directly related takeaway. We ran a version of our analysis to identify how various brands performed.
The graph above summarizes the regression findings. We know that independent hotels improved in quality and on average brands did not. But there’s some interesting differences between brands. This plot shows the erosion of the brands’ quality advantage caused by TripAdvisor penetration on the y-axis plotted against the brand advantage in the absence of TripAdvisor on the x-axis. All the brands below the x-axis saw their advantage eroded, while the brands above the x-axis actually improved relative to independents. Likewise, brands to the right of the y-axis held an initial advantage while brands to the left were initially disadvantaged (in the absence of online reviews). Clearly, most brands are losing ground and some never had ground to lose. But most of the large brands are in the lower right quadrant that we call status-quo premium brands. These are brands that have historically profited from being positioned in the upper echelon within their segment but now face increasing competition from independents on quality because of online reviews. Very very few brands are performing better due to online reviews. These include Ritz Carlton (not labeled) Knights Inn (which could only go up in quality). In any case, for hotel industry folk, this graph represents some interesting tidbits to take away from our research.
But what about beyond the hotel industry? Remember, we hypothesized that these effects are due to 2 factors: quality is uncertain prior to consumption and there is little repeat business to ensure credibility in quality statements. Under these circumstances, online reviews should facilitate quality competition to allow independent companies with low initial externality for poor quality to provide high quality products. I think many companies in the new wave of direct-to-consumer online manufacturer/retailers benefit from this mechanism. For example, one of my favorite furniture stores is Article. It sells relatively high quality furniture (in terms of design, materials, and construction) online. It is not a large brick and mortar furniture chain, furniture quality cannot be easily assessed through pictures (though design quality can), and furniture is an infrequently purchased product, but Article is still able to succeed. I think much of this has to do with the good word of mouth it receives through online reviews and commercial press. I don’t think furniture companies could have imagined launching a successful high end catalog-only store 10 years ago.
So despite being a study of TripAdvisor reviews and hotel quality, I think our study speaks to the role of online reviews on quality competition in a variety of markets. Your voice as a consumer matters! We could use more trustworthy platforms for online reviews to facilitate the production of better quality products and services in a variety of markets. A better RateMyProfessor, perhaps? How about more trustworthy reviews for doctors? There is definitely great social value in investing in trustworthy and easily obtainable measures of quality. Just ask independent hotels and their customers.
Dr. Yang Wang 