# Keeping a 150 to 200 ms call out of a check that needs 14 to 30 ms

PayPal · Nov 2023 – Mar 2024
Author: Henrique Senra (https://henriquesenra.com)
Web version: https://henriquesenra.com/cases/paypal-eds-to-rps-parent-eligibility

> Product teams wanted to decide what shows on PayPal's Home screen by calling their own APIs. I helped set the rule that sends user segmentation through one fast service with attributes computed daily, and the cost is day-old data.

## Key outcomes

- A 150 to 200 ms call kept out of an eligibility check that should answer in 14 to 30 ms.
- Two or three domain teams moved to daily-updated segmentation attributes. One kept its own call after speeding up its API.
- Accepted cost: an attribute can be up to a day stale.
- Slowdown avoided on the app's first screen. No after figure was measured.

Tags: Latency budget, Freshness trade-off, Platform boundaries, Home

## TL;DR

- **Problem.** Domain teams wanted their own data calls to decide who sees their card at the top level of the Home screen. Each call took 150 to 200 ms.
- **My role.** I negotiated and explained the technical case to the Home team, the platform's engineers and two or three domain teams. I measured nothing afterward.
- **Decision and cost.** Parent-level segmentation goes only through the real-time segmentation service, with attributes updated once a day. Users can see day-old eligibility.
- **Outcome.** The slow call stayed out of the eligibility check. Business end: slowdown avoided on the first screen, no before and after figure.

## Context

The PayPal app's Home screen works in two layers. Zones are the containers on the screen, and organisms are the cards that can appear inside each container. For the Financial Products zone, the Home team owned the zones and a parent organism for each domain, and the domain teams owned the child organisms. A parent hides itself when none of its children is available.

Cards in PStudio, PayPal's no-code personalization platform, could pull values from EDS, an external data source that a domain team hosts. The same data sources also served as eligibility checks, joined into small AND and OR expressions called contextual rules.

The domain teams wanted to use EDS to compute eligibility at the parent. The call took 150 to 200 ms, for an answer that should take 14 to 30 ms. That is fast enough to fetch card content, because the dashboard builds its skeleton with the parent and zone call. It is too slow for the eligibility check, which has to resolve before any data is fetched.

The default on Home was zero trust. A card stayed hidden until a platform said it was right to show it. The exceptions were very basic things that 99.99% of users had access to, such as balance.

## What I owned

The conversations involved me for Home, my technical manager on the Home team, the technical manager for PStudio, a developer who did the segmentation work and two or three domain teams. It took a lot of negotiation and technical explanation. I managed to cover some of it and not all. The idea that RPS, the real-time segmentation service, was the answer was already a known limitation inside the PStudio engineering team, so I did not originate it. My part was the negotiation, and translating between the platform's engineers and product managers from other domains.

I did not measure latency or errors after the migration. That was outside my scope.

## Options

```mermaid
flowchart TD\n  Q["Does the parent show for this user?"] -->|"Rejected: call the domain API (EDS), 150 to 200 ms"| X["Too slow before content loads"]\n  Q -->|"Chosen: RPS segment, attribute updated daily"| Y["Fast answer, data up to a day old"]\n  Y --> Z["Other filtering only on child cards"]
```

| Option | Why not | Cost accepted |
| --- | --- | --- |
| Let domain teams use EDS at the parent | It adds 150 to 200 ms in a path that has 14 to 30 ms. | Domain teams that wanted it were unhappy. |
| Keep complex rules at the parent | Same latency problem. | Complex rules moved down to each team's children. |
| A cache or an hourly batch in between | Not considered, because the segmentation service was an obvious choice. | None recorded. |

RPS returns a set of tags for the segments a user belongs to. PayPal built it to answer eligibility very fast, and the domain teams already used it. What they needed were new attributes in it, so they could stop calling their own API as an EDS. I found that the service had enough speed, scalability and reliability to hold the rule at the parent, and I proposed creating the attributes. A batch process, as I remember it, updates each attribute daily from the domain team's external source, since real time was not needed.

The boundary, as shipped. Parent-level user segmentation has to use an RPS segment. Parents can also filter by country, app version and whether the device supports NFC. All other filtering, including experimentation tool checks and EDS, can live only at the child level. Not every domain had to use RPS, only the ones that wanted parent-level filtering. I believe two or three did. One kept using EDS because it fixed its API latency. I think it was the buy-now-pay-later team, and I do not know why they wanted it so much. A real-time attribute that RPS did not support is my guess.

## Results

The outcome is a design rule. The slow call stayed out of the eligibility check, and the cost is day-old data. A stale attribute can cause two problems. An object the user should receive is not sent, or an object the user should not receive is sent. It was not a critical problem.

The only latency figures I have are the call that took 150 to 200 ms and the 14 to 30 ms it should take. I have no number after the change. There was also no business metric tied to it.

You can argue RPS was the right choice from the start and EDS was technical debt that had become the culture and the standard. EDS should also have been split into content and rule sources, but I think that idea came years before I joined and was never properly discussed.

## Friction and what I would change

The domain teams could not see the problem or why the solution was needed. For them, using EDS at the parent already solved it. RPS had no product lead, which complicated everything. PStudio only had technical leadership, which put a barrier between me and the product managers of other domains, and I acted as the translator.

I would invest more time in documentation. I would cut outdated parts, update sections and record video that lasts. PStudio's basic knowledge was always the platform's biggest challenge.

## Limits

- No after figure exists. Do not read this as a measured latency gain.
- The number of domains that moved is my recollection, two or three.
- The batch process is described from memory.
- Names of colleagues stay private.
