A/B test on Android means you ship both variants inside one build, decide at runtime which one a given user gets, log the moment they actually see it, and compare a metric between the two groups. On most Android teams that is Firebase Remote Config plus Analytics, wired together by Firebase A/B Testing, and the hard parts are not the SDK, they are the loading strategy and the statistics.
First, separate three things that get confused in interviews.
- A feature flag answers can we turn this off. It is an operational switch, usually all on or all off, and its job is a kill switch for a broken feature. No measurement required.
- A staged rollout answers is this build safe. It is Play releasing one binary to a growing percentage of users while you watch crash and ANR rates. It is about a version, not about a variant. More on that in what is a staged rollout.
- An A/B test answers which version is better. Two or more variants live at the same time, users are split between them, and a metric decides. Same underlying plumbing as a flag, completely different question.
The mechanism, whatever platform you use.
- A parameter with variants. One key, two or more values, a control and a treatment. Everything else is bookkeeping around that key.
- A sticky assignment. The same user must get the same variant every session, otherwise your groups blur together and the result is noise. Firebase does this by hashing the experiment id together with the Firebase installation id, which is deterministic and survives restarts. Note that installation id means per app install, so a user with a phone and a tablet can land in both arms.
- An exposure event. Log an event the moment the user actually reaches the screen where the variant differs, not when the app starts. If you count everyone who launched the app, you dilute the test with people who never saw either version.
- One metric it is judged on. Decided before you start. Firebase calls it the goal metric and lets you carry up to five secondary metrics alongside it.
Where the assignment happens, and what it costs you.
- Server side, your own backend or a platform. The server knows the user id, so the variant is consistent across phone, tablet and web, and you can change the split without an app release. The cost is a network round trip on a path that may be on the critical rendering path.
- Client side through Remote Config. The SDK caches values on device, so reads are instant and work offline against your baked in defaults. The cost is that the assignment is per install, and a user who has not fetched yet is silently in the default group.
- The honest tradeoff. Cross device consistency and instant changes point at the server, latency and offline behaviour point at the client. Most Android apps pick the client because the login state is not always available at launch.
Firebase A/B Testing, the answer most interviewers are looking for.
- It is Remote Config plus Analytics. You define a baseline and at least one variant on a Remote Config parameter, pick an exposure percentage of the user base, and target with the normal Remote Config conditions, app version, platform, language, country, Analytics audiences and user properties.
- It splits evenly and the weights are frozen. Variants are weighted equally by default, and the weights cannot be changed once the experiment is running, so getting the split right up front matters.
- It reports against one goal. Revenue, retention, crash free users or a custom Analytics event, plus secondary metrics. It uses frequentist inference at a significance level of 0.05 and calls a variant the leader when the difference from baseline is significant.
- It wants two weeks. Firebase names two weeks as the recommended minimum runtime for a typical Remote Config experiment, and results refresh once a day, so there is nothing to watch hourly anyway.
- Membership rides on your Analytics events. Firebase writes experiment and variant membership as user properties on every Analytics event, so you can rebuild the analysis yourself in BigQuery if you do not trust the console.
The Remote Config loading trap, which is the part candidates miss.
- A fetch that activates mid session flips the UI under the user. They are halfway through a screen and the button changes. That is a bad experience and it also corrupts the test, because the person saw both variants.
- Ship defaults in the build. Set them from an XML resource so the app is correct offline and on first launch, before any fetch has ever completed. Covered further in what is Firebase Remote Config.
- Then pick one of three strategies. Activate cached values at startup and fetch in the background for the next launch, which is the safest. Or fetch and activate behind a loading screen with your own short timeout, which Firebase recommends for experiments. Or fetch and activate on launch and accept the risk, which is only fine for changes with no visible effect.
- Real time updates are for config, not for experiments.
addOnConfigUpdateListener pushes new values as soon as they publish, and it saves you from hammering the backend with startup fetches, but activating on that callback is exactly the mid session flip you were avoiding.
- Order matters for the exposure event. It has to fire after the values are activated and before the code branches on them, otherwise you are attributing users to a variant they were not actually served. See also changing parameters without an app update.
// 1. Defaults ship in the build, so the app is correct offline and on first run.
val remoteConfig = Firebase.remoteConfig
remoteConfig.setConfigSettingsAsync(
remoteConfigSettings { minimumFetchIntervalInSeconds = 3600 }
)
remoteConfig.setDefaultsAsync(R.xml.remote_config_defaults)
// 2. Fetch and activate once, at launch, before the first experiment screen renders.
remoteConfig.fetchAndActivate().addOnCompleteListener { showCheckout() }
// 3. Read the variant, then log the exposure at the moment the user sees it.
fun showCheckout() {
val variant = remoteConfig.getString("checkout_button_variant") // "control" or "single_tap"
Firebase.analytics.logEvent("checkout_viewed") {
param("checkout_button_variant", variant)
}
render(variant)
}
Running it properly, which is what a senior answer sounds like.
- One hypothesis, one primary metric. Written down before the test starts. If you cannot say what number has to move and by how much, you are not running an experiment, you are looking at a dashboard.
- Sample size and duration fixed up front. Work out how long you need for the effect you care about, then run for at least that. Always run whole weeks, because weekday and weekend users behave differently.
- Guardrail metrics alongside the goal. Crash free rate, ANR rate, retention, revenue. A variant can win on taps and lose on money, and Firebase deliberately picks the leader on the primary metric alone, so reading the secondaries before rolling out is your job.
- No peeking. Checking daily and stopping when it first looks significant inflates your false positive rate badly. Pick the end date and honour it.
- Watch for novelty effects. A new UI gets extra taps because it is new, and the lift fades. Another reason short experiments lie.
- Hold a small group back. Keeping a few percent of users on the old behaviour after you ship the winner is how you find out months later whether the win was real.
- Delete the flag when you are done. Every finished experiment leaves a dead branch in the code and a stale parameter in the console. Cleaning them up is part of the work, not an optional tidy.
Pitfalls that are specific to mobile.
- Adoption is slow, so your cohort is skewed. A parameter only reaches users who have the build that reads it. Early in a release, that group is disproportionately people who auto update, on newer devices, on better networks. They are not your average user.
- Gate the experiment on app version. Target the minimum version that contains the code, otherwise older installs sit in the control arm forever and quietly poison the baseline.
- Never run an experiment on top of a staged rollout of the same build. If the variant only exists in the new version and the new version is at 10 percent, you are measuring the rollout population, not the variant. Finish the rollout, then start the experiment.
- Cache and offline reads. Users on planes and bad networks run on defaults for a long time. Make the default the control arm so those sessions are at least attributed honestly.
Third party platforms do the same job with more statistical machinery.
- Statsig, LaunchDarkly, Optimizely and Amplitude Experiment. All give you sticky bucketing, automatic exposure logging on the read call, sequential or Bayesian analysis, and flags plus experiments in one system. Amplitude's Android SDK tells you to fetch at startup and wait for the result before rendering, which is the same flicker problem Firebase describes, so the shape of the answer does not change with the vendor.
There is one A/B test on Android that has nothing to do with your code.
- Play Console store listing experiments. You test the icon, the feature graphic, the screenshots and, on a localised listing, the description, against unique user install clicks or open clicks. Play splits store visitors evenly across variants, you choose what percentage of visitors are in the experiment, and the console estimates how long it needs. Run it at least a week for the weekday and weekend mix, change one asset at a time, and note that an experiment ends automatically after six months.
In the room, lead with the distinction, a flag is a switch, a rollout is about a build, an A/B test is about a metric. Then give the mechanism in one breath, a Remote Config parameter with variants, a sticky per install assignment, an exposure event logged at the screen, and one primary metric. Say Firebase A/B Testing by name and that it needs about two weeks. The detail that separates a strong answer is the loading strategy, explain that you activate at startup or behind a short loading screen rather than mid session, because a UI that changes under the user both looks broken and ruins the data. Finish on discipline, one hypothesis, guardrails, no peeking, and delete the flag afterwards.