Context & Problem
Experimentation is the closest thing product management has to a scientific instrument - and it's exactly the tooling most teams outsource. SaaS vendors like Optimizely and LaunchDarkly are excellent, but they come with two structural costs: your users' behavioural data leaves your infrastructure, and assignment decisions depend on an external network call.
For privacy-sensitive teams - or anyone building an internal experimentation platform - that trade is unacceptable. The gap: a self-hosted platform covering the full experiment lifecycle (draft → running → paused → concluded) with assignment that is fast, deterministic, and entirely local.
Design Principles
- Assignment must be a pure function. Same experiment, same user → same variant, forever, with no database lookup and no network call. This is what makes SDKs trivial and results trustworthy.
- Ingestion and analysis are decoupled. Events flow through a buffer (Kafka via Broadway) so analysis load can never slow down the product being measured.
- Statistics are a service, not a library call. Peeking at results mid-experiment inflates false positives; the platform should offer sequential methods with proper alpha-spending, not just a t-test endpoint.
- Multi-tenant from day one. Tenancy, roles (viewer/editor/admin), API keys, and audit logging are platform features, not bolt-ons.
Solution & Approach
1. Deterministic assignment in Rust
The assignment core is 182 lines of Rust exposed to Elixir as a Rustler NIF: MurmurHash3 (128-bit) over "{experiment_key}:{user_id}", mapped into a 10,000-slot basis-points bucket space, then a cumulative-range lookup picks the variant. Stateless, sub-microsecond-scale work, and reproducible from any service that implements the same hash. A pure-Elixir fallback keeps development working where the NIF can't compile.
// assignment_core: bucket = murmur3_128(key) % 10_000
// variants own cumulative basis-point ranges: [0,5000) A, [5000,10000) B
pub fn assign(experiment_key: &str, user_id: &str, ranges: &[Range]) -> usize {
let bucket = murmur3_x64_128(format!("{experiment_key}:{user_id}")) % 10_000;
ranges.iter().position(|r| r.contains(bucket)).unwrap()
}2. Event ingestion built to absorb bursts
An event_collector app receives single and batch events, publishes to Kafka through Broadway, and buffers through Kafka outages. Analysis is pulled by Oban-scheduled workers, never pushed synchronously - the measured product stays fast no matter what analytics is doing.
3. A statistical engine that respects peeking
The Python/FastAPI engine implements z-tests for proportions and Welch's t-test for the classic cases, plus O'Brien-Fleming and Pocock alpha-spending boundaries for sequential monitoring, and a power/sample-size calculator so experiments are sized before they start. Services authenticate to it with an internal key and propagate W3C trace context.
4. Lifecycle, guardrails, and audit as first-class objects
Experiments move through explicit state machines with optimistic locking. Eight background workers handle scheduled starts/ends, analysis triggers, guardrail monitoring, notifications, data retention, and partition management. Every state change lands in an audit log; GDPR export/erase endpoints exist because a data platform without them isn't self-hostable in good conscience.
Implementation
Four Elixir umbrella apps (domain core, Phoenix web layer, event collector, assignment engine wrapper) plus the Rust core, the Python statistical engine, and a React 19 + TypeScript dashboard - 8 runtime components, roughly 8,000 lines of first-party code. The dashboard covers experiment CRUD, lifecycle actions, metric definitions, feature flags, API-key and user administration, and an audit log, with Phoenix Channels pushing live updates to open experiment pages. JWT sessions and API keys are tenant-scoped; tenancy is enforced at row level.
Outcome & Metrics
- 50+ API routes across experiments, metrics, flags, analytics, audit, GDPR, and admin - all tenant-scoped
- Deterministic assignment - MurmurHash3 into 10,000 basis-point buckets, served from a 182-line Rust NIF with Elixir fallback
- 4 statistical methods - z-test, Welch's t-test, O'Brien-Fleming and Pocock sequential boundaries - plus power analysis
- 8 background workers for lifecycle automation, guardrails, retention, and partitioning
- Full lifecycle + audit - draft → running → paused → concluded with optimistic locking and audit trail
- Honest scope: feature-flag creation UI, Bayesian analysis exposure, and CUPED are deliberately deferred; the results cache is still in-memory
- Source: github.com/atavisticrystal6888/A-B-Testing-Platform
Learnings
What Worked
Putting the hash in Rust was less about raw speed and more about making determinism a contract: the bucket math is small enough to re-implement in any SDK language and verify against golden vectors. Decoupling ingestion from analysis via Kafka meant the statistical engine could stay simple - it reads aggregates on a schedule instead of chasing a stream.
What I'd Change
Multi-tenancy at row level keeps deployment simple but pushes discipline into every query - schema-level isolation would trade migration convenience for a harder security boundary. And I'd wire sequential-analysis recommendations into the dashboard sooner: the methods exist, but a PM's real question is "can I stop this experiment today?", and the UI should answer exactly that.