Learning how to build an recommendation system from initial signals

Execution limits: External-service and vendor-runtime examples are reference exercises, not executed deployments. Synthetic numbers are assumptions, not benchmark results.

Building a Targeting System from Early Adopter Signals

Goal: Given a small set of early adopters, build a scoring model to identify users most likely to adopt a product.

Git Repo: github.com/dinesh-coderepo/targetting-system

🔑 Key Concepts at a Glance

🏗️ System Architecture

The business decision is whom to contact next, not which item to show an existing customer. Follow the early-adopter labels into training, then follow the resulting scores to two different destinations: a ranked outreach list and an evaluation on held-out outcomes. Known adoption status is useful supervision, but the features must describe the user before that outcome—not behaviour caused by already adopting.

🔧 Background & Prerequisites

1. Types of Recommendation Systems

This taxonomy places the targeting problem beside the wider recommendation field; it is not a list of components to implement before starting. Deep learning is an implementation family that can be used for collaborative, content-based or hybrid models, not a mutually exclusive source of recommendation evidence. Collaborative filtering needs interaction patterns that a new product may not yet have, which is why the worked example below starts with user features and a classifier.

A hybrid can combine interaction and content evidence; the choice needs measured coverage and quality rather than a universal product attribution.

2. The Cold Start Problem

This is the core challenge for this blog — very few adopters means extreme data sparsity. In the branches below, user cold-start methods help when a person has little history, while item cold-start methods help when a new item has little exposure. For early-adopter targeting, start with the lookalike branch; similarity can suggest candidates, but it does not show that contacting them will cause adoption.

Solutions for targeting with few adopters:

• 🔹 Feature similarity — Match non-adopters against adopter feature profiles
• 🔹 Lookalike modeling — Find users who "look like" early adopters (demographics + behavior)
• 🔹 Response scoring — Binary classifier: adopted within the mature outcome horizon (1) vs not observed to do so (0)

3. Propensity / Targeting Model

The heart of this project is ranking eligible, not-yet-adopted users by a defined future outcome. Here “propensity” means adoption response, not the causal-inference propensity score P(treatment assigned | covariates). Neither is itself an estimate of incremental treatment effect. A model output in [0,1] needs calibration checks before it is interpreted as an adoption probability.

Feature Categories:

Model Choices:

⚠️ Class Imbalance: If only 1% are adopters, an always-negative baseline has 99% accuracy while finding none of the adopters. Compare a baseline and decision metrics first; training-only weighting/sampling changes calibration and must be evaluated. SMOTE or focal loss is not mandatory.

4. Evaluation Metrics

⚠️ Do not use accuracy alone with imbalanced data; compare a majority baseline and decision-relevant precision/recall and cost.

⚠️ Use time-based evaluation when predicting future behavior. A random split is not inherently invalid for every question, but it does not establish future-period performance or remove availability-time leakage.

5. Tools & Libraries

Next steps

Work through collaborative filtering, factorization, propensity, features, imbalance, metrics, cold start, model comparison and the Monolith connection below. Optional third-party training is a reference procedure, not an executed result.

🔧 Reference Implementation — Propensity Model with Lookalike Scoring

The corrected reference uses mature prediction-time labels, a chronological holdout, and one fitted preprocessing pipeline reused at scoring. It replaces the random split and separately recreated one-hot columns in the original scaffold. Dependencies are not installed here, so this API-shaped example is syntax-checked, not runtime-verified. Treat a negative as “not observed to adopt within the fixed outcome horizon,” not “will never adopt.”

# targeting.py — reference API shape for pandas + scikit-learn 1.5.
# Not executed in this review: those optional dependencies are absent.
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, average_precision_score
 
NUMERIC = ["age", "logins_30d", "features_used", "tenure_days", "team_size"]
FEATURES = NUMERIC + ["industry"]
 
def temporal_split(rows, cutoff, evaluation_end):
    # Each row is a prediction-time snapshot, not today's user profile.
    # label_available_at records when its fixed-horizon outcome is knowable.
    data = rows.copy()
    for col in ["prediction_time", "label_available_at"]:
        data[col] = pd.to_datetime(data[col], utc=True)
    cutoff = pd.to_datetime(cutoff, utc=True)
    evaluation_end = pd.to_datetime(evaluation_end, utc=True)
    train = data[(data.prediction_time < cutoff) &
                 (data.label_available_at <= cutoff)]
    test = data[(data.prediction_time >= cutoff) &
                (data.prediction_time < evaluation_end) &
                (data.label_available_at <= evaluation_end)]
    if train.empty or test.empty or train.label.nunique() != 2:
        raise ValueError("need training classes and a mature held-out population")
    return train, test
 
def train_model(train, test):
    numeric = Pipeline([("impute", SimpleImputer(strategy="median")),
                        ("scale", StandardScaler())])
    categorical = Pipeline([
        ("impute", SimpleImputer(strategy="most_frequent")),
        ("encode", OneHotEncoder(handle_unknown="ignore", sparse_output=False))])
    preprocessing = ColumnTransformer([
        ("numeric", numeric, NUMERIC),
        ("industry", categorical, ["industry"])])
    model = Pipeline([("features", preprocessing),
                      ("classifier", LogisticRegression(max_iter=1000))])
    # Vocabulary, imputation and scaling fit on training rows ONLY.
    model.fit(train[FEATURES], train.label)
    scores = model.predict_proba(test[FEATURES])[:, 1]
    metrics = {"test_rows": len(test), "positive_rate": float(test.label.mean())}
    if test.label.nunique() == 2:
        metrics.update(roc_auc=roc_auc_score(test.label, scores),
                       average_precision=average_precision_score(test.label, scores))
    else:
        metrics["limitation"] = "single-class holdout: ranking metrics not reported"
    return model, metrics
 
def score_and_rank(model, eligible_snapshots, top_k=1000):
    if type(top_k) is not int or top_k < 1:
        raise ValueError("top_k must be positive")
    data = eligible_snapshots.copy()
    # Caller establishes consent, eligibility, tenant scope and feature availability.
    data["score"] = model.predict_proba(data[FEATURES])[:, 1]
    return data.sort_values(["score", "user_id"], ascending=[False, True])[
        ["user_id", "score"]].head(top_k)

Evaluating with a Proper Time-Based Split

Split by the timestamp when a prediction would have been made, not merely by signup date. All features must reflect data available then, and training labels must have matured before the training cutoff. The code deliberately requires prediction_time and label_available_at; it cannot infer those historical facts from a current profile table. If repeated snapshots of one user appear, evaluate whether group separation is also required by the deployment question.

Avoid outcome-dependent maturity: for a seven-day horizon, even a user who adopts on day one should have a label-availability boundary no earlier than the end of the same horizon (plus the declared reporting delay). Otherwise recent positives can enter evaluation while equally recent non-adopters are excluded, inflating the observed response rate. For T=1 June 00:00 UTC, the interval ends at 8 June 00:00 UTC; evaluation on 5 June must not include that snapshot in a seven-day cohort. Use a fixed prediction window whose complete outcome horizons have elapsed, then account for missing/late reports explicitly. The helper filters supplied timestamps; it cannot prove the supplied history, consent, absence of missing events or validity of that maturity contract.

The test set is not used to fit imputation, scaling or the category vocabulary. Unknown categories map through the fitted encoder rather than creating a different feature-column layout. Compare the baseline with nonlinear models using the same time-aware protocol; no model is declared inherently more accurate. Inspect calibration separately before treating a score as a reliable probability.

For the logistic baseline, p = 1/(1+exp(-z)) with z = intercept + dot(weights, transformed_features). Odds are p/(1-p) = exp(z). Holding other transformed features fixed, increasing feature j by one unit multiplies the model odds by exp(weight_j); with StandardScaler that is one training standard deviation of the imputed numeric feature, not one raw year or login. It is an association under the fitted model, not a causal effect of changing age or usage.

Lift Chart — The Right Way to Present Results

Lift translates a score ranking into the targeting decision. If 2% of a held-out population adopts, but 8% of its highest-scoring tenth adopts, that group has 4× lift. The function below computes that ratio from observed labels; it does not measure the effect of sending a campaign, and tied scores or small cohorts make the deciles less stable.

import math
 
def lift_chart(y_true, y_score, deciles=10):
    if len(y_true) != len(y_score) or not y_true:
        raise ValueError("nonempty aligned labels and scores required")
    if type(deciles) is not int or not 1 <= deciles <= len(y_true):
        raise ValueError("invalid bucket count")
    if any(y not in (0, 1) for y in y_true):
        raise ValueError("binary observed labels required")
    if any(not math.isfinite(float(p)) for p in y_score):
        raise ValueError("finite scores required")
    n = len(y_true)
    base = sum(y_true) / n
    # Deterministic ties by input index; bucket 1 is the highest-scoring group.
    order = sorted(range(n), key=lambda i: (-y_score[i], i))
    result = []
    for bucket in range(deciles):
        indices = order[bucket*n//deciles:(bucket+1)*n//deciles]
        rate = sum(y_true[i] for i in indices) / len(indices)
        result.append({"bucket": bucket+1, "n": len(indices), "rate": rate,
                       "lift": rate/base if base else None})
    return result
 
assert lift_chart([1,1,0,0], [.9,.8,.2,.1], 2)[0]["lift"] == 2
assert lift_chart([0,0], [.9,.1], 2)[0]["lift"] is None
assert sum(x["n"] for x in lift_chart([1,0,0], [.5,.5,.5], 2)) == 3
print("lift chart: ties, aligned populations and zero-base-rate checks PASS")

This function accepts Python sequences such as lists, not a pandas Series/NumPy array with ambiguous boolean truth. Deciles are disjoint rank buckets; they are not cumulative top-k metrics. With three rows and two buckets, sizes are one and two, so compare their denominators rather than assuming equal-sized groups. Integer options reject booleans explicitly because Python otherwise treats True as the integer one.

Acceptable lift depends on base rate, intervention cost, uncertainty and the actual business decision. Neither 3× nor 1.5× is a universal quality threshold.

Propensity is not incremental treatment effect

A propensity/response model ranks likelihood of a defined outcome under observed data. It does not identify who would change behavior because of an intervention; uplift requires causal assumptions and suitable treatment/control evidence. Sampling existing adopters as positives and treating every other user as a negative can create label-delay and selection biases. Define eligibility, observation window, outcome horizon and exposure before fitting.

A time split must also keep preprocessing, imputation, category vocabulary and feature construction inside the training boundary. Exclude post-adoption features and information arriving after the prediction time. Evaluate precision/lift at the actual contact budget, calibration and cohort coverage, and compare against a simple popularity/base-rate baseline. A lift chart needs its denominator: top-decile conversion divided by population conversion in a compatible held-out population. If the population rate is zero, lift is undefined. The reference implementation is a synthetic learning example and not evidence of advertising effectiveness or lawful audience targeting.

Failure exercise and worked answer

A user adopted on Friday, and a feature counts usage through Sunday. Can it predict Friday adoption in a Monday-trained historical dataset?

Sources and review boundary

Primary passages were checked on 2026-10-07. The versioned sources below support the mechanisms discussed. No cluster, cloud service, load benchmark, provider send or model inference was executed for this review.

• Google recommendation systems overview — Recommendation architectures separate candidate generation, scoring and reranking.

• Feast point-in-time joins — Point-in-time retrieval scans backward relative to entity timestamps; created-time filtering is a separate supported option where available.

• scikit-learn 1.5.2: common pitfalls — fit preprocessing on training data only and reuse the fitted pipeline.

From interactions to candidates: collaborative filtering worked through

Targeting one product is not the entire recommendation problem. To see the distinction, use three items A, B, C and three users: u1 consumed A and B; u2 consumed A and C; u3 consumed B and C. Treat these as binary observed interactions, not ratings of unseen items. Each pair of users has cosine similarity 1/2: their dot product is one and each norm is square root of two. A user-neighbor score for u1's unseen C is (0.5×1 + 0.5×1)/(0.5+0.5)=1. This recommends C from both neighbors, not from an adoption propensity label.

Item-based CF instead represents A as (1,1,0), B as (1,0,1) and C as (0,1,1). C has cosine 1/2 to A and B; aggregating similarities to u1's consumed items gives 1. For large data, retain a bounded neighbor list and evaluate approximate retrieval against an exact sample. “Doesn't scale” is a cost warning, not a theorem that user-based CF cannot be served.

from math import sqrt, isclose
 
interaction = {"u1": {"A", "B"}, "u2": {"A", "C"}, "u3": {"B", "C"}}
 
def cosine_sets(left, right):
    return len(left & right) / sqrt(len(left)*len(right)) if left and right else 0.0
 
def user_candidates(user, history):
    known = history.get(user, set())
    totals, weights = {}, {}
    for other, items in history.items():
        if other == user:
            continue
        sim = cosine_sets(known, items)
        if sim <= 0:
            continue
        for item in items - known:
            totals[item] = totals.get(item, 0) + sim
            weights[item] = weights.get(item, 0) + sim
    return sorted(((item, totals[item]/weights[item]) for item in totals),
                  key=lambda row: (-row[1], row[0]))
 
assert isclose(cosine_sets({"A","B"}, {"A","C"}), 0.5)
assert user_candidates("u1", interaction) == [("C", 1.0)]
assert user_candidates("new", interaction) == []
print("collaborative candidates: C for u1; cold-start explicitly empty")

This binary positive-neighbor fixture gives tied scores whenever every contributing neighbor consumed the candidate. Real CF can use observed rating deviations, confidence weights or richer interactions; do not confuse this simple voting score with a calibrated probability. Cold-start returns no evidence, allowing an explicit metadata/popularity fallback rather than silently pretending the new user has a learned vector.

Matrix factorization: what the SVD family learns

An explicit-rating factor model predicts mu + user_bias + item_bias + dot(user_vector,item_vector). Fit only observed training ratings; a missing rating is not zero. Minimize squared error plus regularization on biases/factors. The Surprise library's SVD is a regularized learned factor model, not simply a dense textbook SVD of a matrix filled with zeros. Its biased prediction and stochastic-gradient updates are specified in the primary API documentation.

A worked gradient makes the objective concrete. With target rating 4, prediction 3, scalar user factor 0.2, item factor 0.5, learning rate 0.1 and regularizer 0.01, error is 1. The simultaneous factor steps are p'=0.2+0.1×(1×0.5−0.01×0.2)=0.2498 and q'=0.5+0.1×(1×0.2−0.01×0.5)=0.5195. The second uses the old p, not the updated p. A regularized bias step similarly adds learning_rate×(error−regularizer×bias). Unknown IDs need a baseline/content fallback; factorization does not invent an informative vector from an ID never observed.

Complete optional Surprise procedure, not executed: construct (user_id,item_id,rating,prediction_time) observations with a documented rating scale; split chronologically at deployment time; fit vocabulary/baselines/factors only on training observations; tune factor count/regularization on a later validation window, not the final test; evaluate RMSE/MAE for rating prediction and ranking metrics only over a defined eligible candidate set; separate known/new users/items; compare to global and item-mean baselines; persist fitted model, feature/vocabulary versions and training cutoff; score only authorized candidates, removing known-consumed items when the product requires discovery. Installation, third-party training and data downloads are deliberately not performed in this review.

Optional Surprise implementation — reference only

This implementation makes the factorization procedure concrete without downloading a dataset or claiming a training result. Supply an already authorized pandas frame ratings with string user_id/item_id, finite ratings in [1,5], and UTC prediction_time/label_available_at. Each row is an observed rating outcome at its declared prediction time; repeated events and candidate eligibility need a task-specific policy. The fixed cutoff and evaluation end must satisfy the same complete-horizon/availability contract as the classifier. These calls follow the Surprise getting-started API; pin and verify the selected release before running. Neither pandas nor Surprise is installed or executed for this reference review.

# Optional API reference; NOT RUN. No load_builtin or dataset download.
import pandas as pd
from surprise import Dataset, Reader, SVD, accuracy
 
def fit_rating_reference(ratings, cutoff, evaluation_end):
    rows = ratings.copy()
    for col in ["prediction_time", "label_available_at"]:
        rows[col] = pd.to_datetime(rows[col], utc=True, errors="raise")
    cutoff = pd.to_datetime(cutoff, utc=True)
    end = pd.to_datetime(evaluation_end, utc=True)
    train = rows[(rows.prediction_time < cutoff) &
                 (rows.label_available_at <= cutoff)]
    test = rows[(rows.prediction_time >= cutoff) &
                (rows.prediction_time < end) &
                (rows.label_available_at <= end)]
    if train.empty or test.empty:
        raise ValueError("need mature train and held-out rating rows")
    data = Dataset.load_from_df(train[["user_id", "item_id", "rating"]],
                                Reader(rating_scale=(1, 5)))
    # Full trainset means the supplied TRAIN partition, not all available rows.
    trainset = data.build_full_trainset()
    model = SVD(n_factors=20, n_epochs=20, random_state=0)
    model.fit(trainset)
    triples = list(test[["user_id", "item_id", "rating"]].itertuples(index=False, name=None))
    predictions = model.test(triples)
    return model, {"rmse": accuracy.rmse(predictions, verbose=False),
                   "mae": accuracy.mae(predictions, verbose=False)}

Twenty factors/epochs are illustrative settings, not tuned recommendations. RMSE/MAE evaluate observed ratings only; they do not measure the ranking of all unseen items. Report known/new-ID cohorts separately: Surprise's biased SVD uses zero unknown biases/factors, so its prediction for an unseen ID is a baseline, not a learned personal vector. Keep the identity types consistent at prediction time. Do not fit on a test set just to make every ID known.

A complete early-adopter dataset contract

For every eligible user at prediction time T, reconstruct features available by T. Define a seven-day adoption horizon [T,T+7 days), then wait until outcomes are mature before labelling; record ingestion/availability time as well as occurrence time. A missing event can be a delayed outcome, not a negative. Training rows need label_available_at ≤ training cutoff; held-out rows need maturity by the evaluation end. Keep consent, tenant and eligibility separate from statistical features. Demographic inputs can proxy protected characteristics; require a justified permitted use and cohort evaluation instead of assuming every available column belongs in a marketing model.

Use the reference temporal_split, train_model and score_and_rank functions above with rows that satisfy this contract. Numeric imputation/scaling and the category vocabulary fit on training only. The caller must exclude existing adopters and ineligible users before scoring. score is initially a model score, not a verified adoption probability; evaluate calibration on an untouched deployment-like population.

Class imbalance is not solved by blindly synthesizing rows. First report the positive count and horizon, then compare an unweighted logistic baseline, appropriate class weights, and a nonlinear challenger under the same split. Apply any sampling inside training folds only. Weighting/sampling can alter calibration; evaluate or recalibrate with mature, representative labels. SMOTE can create implausible combinations in mixed/temporal data and is not mandatory. Focal loss is another training objective, not a drop-in guarantee for a scikit-learn logistic pipeline.

Compare models without inventing results

This is a comparison protocol, not a fabricated leaderboard. For each fitted candidate record test dates, positive count, eligible population, precision/recall/lift at contact budget K, PR/ROC metrics where defined, calibration, cohort uncertainty, and serving cost. Select on validation, lock the decision, then report the final holdout once. Model size and labels determine whether neural networks are justified; they are not automatically the winner.

Cold-start fallback and the Monolith connection

For a new user with no interactions, ask a small set of optional preference questions or offer diverse popular eligible content. For a new item, use verified metadata/content features and a bounded exploration policy to gather evidence. Track fallback exposure separately so training can distinguish model selection from exploration. A low-data adopter classifier can use pre-outcome features but still has estimation uncertainty; gather more labels or abstain rather than extrapolating unsupported certainty.

Monolith solves the dynamic identity/storage and online-training problem at a different layer. It retains separate embeddings, filters rare IDs, expires stale IDs, joins serving features with delayed labels and synchronizes parameters. Those mechanisms do not turn adoption propensity into causal uplift. A randomized outreach/control experiment estimates the intervention's incremental effect under its assignment and outcome assumptions; observational likelihood alone cannot identify people persuaded by contact.

Budget metrics and causal interpretation — worked exercise

Suppose 100 eligible held-out users contain 10 observed adopters. At K=10 the model selects 4 adopters. Precision is 4/10=0.4, recall is 4/10=0.4 and lift is 0.4/0.1=4. Selecting ten users uniformly has expected one adopter, but an individual random selection need not contain exactly one. If K rises, recall cannot fall for a nested ranking, while precision and lift can. With zero population positives, lift and recall are undefined; with no positive-negative pairs, ROC AUC is undefined. A perfect ranking on four positives still carries large uncertainty about future cohorts.

For outreach, consider a randomized test with 1,000 treated and 1,000 control users and adoption rates 0.08 and 0.06. The estimated absolute treatment effect is 0.02, or 20 additional adopters per 1,000 contacts in this sample. It is not the 0.08 response rate. Estimate uncertainty and inspect assignment/interference before a causal conclusion; neither this arithmetic nor the lift program is a real campaign result.

For an independent rank-metric calculation, take ordered binary labels [0,1,1,0]. Among four positive-negative pairs the positive outranks a negative twice, so ROC AUC is 0.5. At k=2 there is one hit: precision=0.5 and recall=0.5. Binary DCG@2 is 1/log2(3); ideal DCG@2 is 1+1/log2(3), so nDCG@2 is about 0.38685. Tied scores contribute half a pair for AUC, while this lift routine breaks ties by input index; declare that convention because top-k membership can depend on it. These numbers show why a full-ranking statistic cannot replace the contact-budget metric.

Full worked practice answers

1. Friday adoption, Sunday feature: it leaks for a pre-Friday prediction. Reconstruct the feature using occurrence and availability times no later than the prediction, and mature labels later; changing only the split date cannot repair leakage.
2. Unseen industry at scoring: the fitted encoder handles it with the trained output schema. Rebuilding dummy columns on scoring data would silently change dimensions/meaning.
3. Only one test class: do not print a ROC AUC. Report the class count, budget outcomes and the inability to measure pairwise discrimination; collect a meaningful evaluation cohort rather than injecting positives into the test.
4. Top decile 1.5× lift: it can be economically useful or useless depending on base rate, incremental effect, contact cost and uncertainty. There is no universal 3× lift threshold for a useful targeting model.
5. New user CF gives no candidates: use the explicit authorized metadata/popularity route and record its provenance. Do not give an arbitrary factor-vector score the label “personalized.”

Sources: Google recommendation stages, scikit-learn 1.5 pitfalls, Surprise factorization, Feast point-in-time joins, and Monolith v2, checked 8 October 2026. Only the standard-library CF and preserved lift program are executed. The pandas/scikit-learn and Surprise paths are complete reference procedures, syntax/documentation reviewed but not trained here.