Introduction and Motivation
At Uber Eats, we provide advertising tools for merchants to help them grow their businesses and boost sales. Our ads delivery system seeks to show the most relevant ads to customers in order to maximize returns for merchants. To improve the system, we need to explore various system configurations. For instance, we need to understand: what is the effect on customers when we increase the number of ad placements? How do customers react to different ad ranking algorithms? Will ads perform better if we modified our bidding algorithms?
We typically measure these changes using experiments. However, they come with challenges. While online experiments can yield precise results, they are costly in terms of time and resources, and pose a risk to hurting business performance. On the other hand, offline analysis using observational data is quicker, but biased.
To address the challenges of these traditional methods, we developed an Ads Simulator tool. This simulator replicates the outcomes of various counterfactual scenarios of the ads system and produces reliable results at a fraction of the cost and time of online experimentation. It is flexible, as it can be (simultaneously) applied to multiple changes such as ads placement design, auction mechanisms, and bidding strategies. Additionally, the simulator can emulate extreme cases that would be challenging to execute in the real world.
Furthermore, the simulator improves engineering efficiency. As the simulator shifts much of the exploratory phase offline, the engineering team can focus on implementing the most promising strategies.
Architecture
Ads Auction and Bidding Flow: Context and Overview
To understand the design for an ads simulator, we first explore a stylized version of the current ads auction and bidding flow. This flow is made up of 6 essential steps that work together to drive successful ads campaigns (Figure 1):
- User Ad Request: The system initializes the flow upon receiving ad requests from the customer. User-contextual information, such as customer profiles, location, time of the day, is passed to the system with the request.
- Retrieval of Candidate Campaigns: Once the context is established, our ads system retrieves a set of candidate campaigns (i.e., bidders) eligible to show under the current context. For instance, if a customer’s search query is related to Thai, the system selects campaigns from advertisers related to Thai cuisine.
- Real-Time Bidding: The campaigns engage in real-time bidding based on the auction context and their real-time campaign state. The system optimizes performance for each campaign by evaluating factors like pacing and budget, and bidding accordingly.
- Auction: The auction module processes the submitted bids and determines the winners and prices. Winners of the auction are granted ad placements.
- Outcomes and Performance Tracking: Customers interact with the app and generate Ads impressions, clicks, and orders. These outcomes provide insights into campaign performances, allowing the system and advertisers to gauge the effectiveness of ads strategies.
- Campaign Performance Monitor and State Update: Customers’ interaction and the passing of time may trigger campaigns to update internal states. This mechanism allows campaigns to adapt to changing market conditions.
Ads Simulator: Overall Architecture
Our Ads Simulator meticulously replicates the ads auction and bidding flow, yielding outcomes from potential ad strategies that mirror the dynamics of our actual ads marketplace. Unlike many simulators that assume auction and bid sequences operate independently, ours captures the real-world intricacies, especially the complexities driven by budget pacing. Moreover, instead of depending solely on synthetic data or a restricted advertiser set, our tool leverages the full array of advertisers in a given market, such as an entire city.
The architecture’s epicenter, depicted in Figure 2, features the Core Ads Flow Simulator. This robust engine manages recurring auctions and bids for every advertiser in the market. Accepting inputs from the Simulation Configuration and Auction Context, it processes interactions and produces comprehensive simulation data logs. Furthermore, our Ads Simulator offers an Analysis Package for a thorough assessment of simulation results. The following sections will detail each module.
Configuration
Ads Simulator can be tailored to probe diverse ad strategies: from bidding and pacing methods, to auction mechanisms and pricing models. Modifying or adding a few parameters suffices for most customizations, facilitating a straightforward configuration.
Auction Context
The Auction Context contains a list of auctions, along with bidders and user details. The Ads Simulator uses this list to replay auctions in order, simulate results, and adjust future auctions if necessary. This list can come from real auction logs or synthetic data.
Real auction logs record all ads candidates at the request level, capturing their bids, contexts, and expected engagement rate. This allows us to see and evaluate ads that might have been filtered by later ranking stages. It addresses the common selection bias in big online ads or recommendation systems. Using this data enables more accurate responses from the counterfactual marketplace, considering complete insights rather than just previous outputs. Meanwhile, synthetic data gives the simulator the freedom to test various market conditions.
Core Ads Flow
The core ads flow is designed to accurately replicate the interactions of the entire marketplace in the production environment. It consists of 3 essential components:
- Ad Campaigns: Each AdCampaign represents a real-world ads campaign from the production environment. The interactions between an AdCampaign and the marketplace, such as bidding, observing outcomes, and adjusting bids for subsequent rounds, are regulated by 3 agents–namely the Bidding Agent, Pacer, and Campaign Monitor. Both Bidding Agent and Pacer employ a reinforcement learning framework to generate bids based on learned policies and campaign states, while Campaign Monitor records campaign state transitions that depend on bidding outcomes (Figure 3).
- Auctioneer: The Auctioneer replicates the actual auction mechanism, including eligibility filtering, scoring, ranking, and pricing. The Auctioneer is designed to be highly flexible to facilitate the testing and exploration of different auction mechanisms.
- User Response Simulator: The User Response Simulator simulates customer interaction with winning ads. This part takes a probabilistic approach and draws from predicted probabilities (e.g., impression probability, click probability) based on contextual information. The simulated responses are then used (along with the simulated auction prices) to calculate business metrics such as ads spend and campaign ROI. We also calibrate predicted probabilities using empirical data to ensure that the simulated metrics closely match real-world outcomes.
Logging and Analysis
The Logging module captures simulation data, including auction outcomes, AdCampaign states, auction paths, marketplace metrics, etc. The Analysis Package provides a wide range of built-in functions and packages for metrics calculation, advanced visualization, and statistical analysis. Both modules are highly extensible, allowing developers to integrate customized logging mechanisms and analysis functions. Scientists and engineers can thus gather rich simulation data and derive valuable insights into ads strategies and marketplace dynamics.
Use Cases at Uber
The Ads Simulator offers valuable insights by accurately replicating the dynamics of the ads marketplace under new ads strategies. Its accuracy and reliability are demonstrated by the high degree of alignment between its simulation results and real-world experimentation results. Engineers and scientists at Uber should be confident to leverage the simulator to make informed decisions, optimize campaign performance, and discover new avenues to innovation.
In this section, we present two use cases that highlight the simulator’s accuracy and capabilities. Both use cases have been validated by online experimentation. They also demonstrate how the simulator provides actionable insights that drive business success.
Use Case 1: Reserve Price Design in Auction Mechanism
A use case of the Ads Simulator is to simulate different reserve price designs. Traditionally, testing various reserve prices involves a time-consuming trial-and-error process with real public traffic, which may result in less ideal business outcomes.
Figure 4 presents the simulation results for searching for the optimal reserve price design. The simulation illustrates the complexity of the task, highlighting that a traditional online experimentation search would be lengthy. The associated implementation cost can be significant with material impact on ads revenue and advertiser outcomes.
Notably, the experimentation results for the chosen design align closely with the simulated results. These changes are so significant that they can rarely be extrapolated using offline analysis, relying on assumptions from historical data. By using this simulator, we prevent the high costs and unreliable results associated with traditional methods, enabling the team to carefully evaluate different designs and select the most effective strategy—all without risking real-world revenue or performance.
Use Case 2: Budget Allocating for Ad Campaigns
Another use case is to help the platform identify the optimal geographical regions for allocating campaign budgets. By determining these optimal regions, we can better guide advertisers on where they might get the most impact from their campaigns. It’s crucial for the marketplace to find the best match between customers and ads to provide a personalized customer experience. A focus on the right regions not only maximizes an advertiser’s campaign effectiveness but also ensures that customers are presented with more personalized ads tailored to their preferences and location. Consequently, when allocating or suggesting increased budgets, we need to ensure that we’re recommending the most effective regions to the advertisers. To do this, we need to understand the outcome of increasing budgets in different geographical regions.
If we are to address this question through experimentation, we need to introduce random variations across various geographical regions. The standard options are impractical: modifying the actual budgets for all campaigns in each region would necessitate unscalable coordination with merchants; providing add-ons on top of the budget would be too expensive for the platform; and relying on budget wiggling experimentation is not a sustainable way of gathering results.
On the contrary, extrapolating impact from observational data runs the risk of selection bias. Additionally, if increasing budgets in a geographical location leads to a change in market equilibrium, observational data would mask this effect.
The Ads Simulator overcomes such limitations and generates reliable data for assessing the potential outcomes of budget scaling strategies. To validate the use of the simulator, we ran an online experiment that scales the budget for each geographical location along with a corresponding simulation that scales the budget by the same ratio. Figure 5 shows the resulting simulated Ads Spend effect and the experimental effect, with each point representing one geographic region. They appear to be well aligned.
Conclusion
We’ve detailed both the challenges and opportunities associated with more efficient and economical evaluation methods for Uber Eats Ads, leading to the introduction of the Ads Simulator. This tool has been seamlessly integrated into the development cycles of our scientists and engineers. It has proven instrumental in guiding and expediting product development, thereby driving our business growth.
Despite its success, there are areas for future improvements. For instance, we can extend the data collection framework by incorporating advertisers which are filtered out during the system’s early retrieval stages. This would broaden the simulator’s applicability to additional ad strategies, such as targeting. Ultimately, refining the user response simulator module—with improved calibration and sampling methods—will ensure that our simulations provide even more precise and reliable results.
Acknowledgements
Special thanks to Ludovica Rizzo and Solomon Barkley for their contribution on enhancing the simulator. Also thanks to Brian Lee and Bo Lin for providing data insights and results on our two successful use cases. The gratitude also goes to Ashish Gupta, Yuriy Bondaruk, and everyone on the Ads Delivery Science and Engineering Team for their insightful feedback and unwavering support. I am deeply grateful to James Lee, Erjie Ang, Guillaume De Roo, Jun Yan, and Ziwei Liu for their continuous support and meticulous review and editing. The success of the ideas in this blog post is a testament to the collective efforts of these remarkable individuals and teams.
