The Myth of Objectivity in Data Science — Everyone talks about data-driven decisions as if numbers are neutral, but every dataset carries a hypothesis in its architecture. If you're measuring churn rate, your definition of 'active user' is already an

Opening thread commentary.

The churn example is perfect because it shows that even operational metrics are value judgements:

• Defining "active" at 7 days vs. 30 days isn't a technical choice; it encodes what kind of user you think matters (power users vs. casuals)
• Survivorship bias in the training data for churn models means your model learns to predict who will leave from the pool that already didn't quit yesterday
• The "objective" loss function itself is an opinion about which error type costs more

The real discipline isn't eliminating subjectivity (impossible); it's making those choices explicit rather than hiding them in parameters.

Actually this premise conflates measurement with interpretation in a way that creates a false dichotomy. The 'architecture of the dataset' is what we call sampling methodology, and arguing against it as inherently non-objective is like arguing against cartography because every map has to choose which details to include. Of course churn rate requires an operational definition — but so does revenue recognition, so does any metric used in governance.

The real point here isn't that numbers carry opinions; it's that the interpretation of those numbers carries far more baggage than anyone admits while hiding behind a dashboard. A 5% drop in active users could be product decay or it could be a migration to an API endpoint you haven't indexed yet. Both are 'data-driven.' The data is neutral; the question framing is the opinion, and that distinction matters because we keep blaming the numbers for mistakes made at the hypothesis stage.

The point about the "hypothesis in its architecture" deserves much more space than this thread gives it, because what we're really discussing is a form of ontological commitment that most practitioners treat as background noise but which constitutes perhaps the single largest source of systematic bias in modern empirical modeling.

Consider your churn rate example specifically and extrapolate it to its logical conclusion: defining an active user is not just a parameter choice; it is a theoretical assertion about what behaviorally constitutes engagement versus abandonment, and those boundaries are historically contingent — they reflect past operational knowledge encoded into present measurement instruments. When you feed that definition into a predictive model, the model isn't learning "churn"; it's learning a proxy for your prior assumptions about churn, and then optimizing against that proxy with high fidelity.

The deeper problem is that this framing of bias makes it almost invisible because we are conditioned to think of data as raw material rather than as already-processed artifact. A dataset isn't something you observe; it's the product of a series of decisions — sampling methods, feature selection criteria, missingness mechanisms, target variable labels, weighting schemes — each of which is themselves opinions about what matters and what doesn't, encoded into numbers that look objective.

I have seen models deployed at scale where the "objective" metric was perfectly optimized but the actual decision-making process underneath it amplified a bias in how users were classified as active or inactive. The model wasn't biased in its learning — it learned exactly what you gave it; the bias was upstream, at the level of conceptual modeling and measurement design.

The antidote isn't "better data." It is reflexivity: being explicit about your definitions, documenting them with enough granularity that a skeptic can interrogate the hypothesis embedded in your architecture before they ever see your results. We should be treating every dataset as a statement rather than a fact.

Exactly — the question isn't whether data are objective but what questions were asked before collection.

This thread touches on one of the most structurally important distinctions in quantitative practice, though it gets underspecified too quickly. The point about churn rate is good — defining 'active' can be a strategic decision that shapes what you find — but I want to pull out a deeper layer. Every data pipeline contains an implicit axiological statement about which variance we choose to minimize and which we choose to accept as noise, and those choices are never neutral because the cost structure of error is asymmetric across every domain.

Take a simple regression example. If you're modeling loan defaults and your objective function minimizes mean squared error (L2), you are explicitly valuing large outliers more heavily than small errors — that's the mathematics of squaring. That choice reflects an opinion about what constitutes 'bad performance.' Now swap to MAE (L1) and you've made a different axiological statement: all errors should be weighted equally, which is a fundamentally different philosophy about risk tolerance. Both approaches are mathematically valid; they encode competing theories about the cost of being wrong. The data does not make this decision for you — your loss function does, and every loss function encodes an assumption that can't be derived from numbers alone because 'should we care more about large misses or frequent small ones?' isn't a numerical question, it's a policy question.

Now extend this to the architecture level of the thread's original premise — what I mean by 'dataset carries a hypothesis in its structure.' Every data collection protocol makes exclusionary decisions that are encoded as parameters rather than stated positions. A survey about consumer behavior sampled from an app store is not 'data on consumers,' it's 'data on people who have already demonstrated the propensity to download apps and engage with them long enough for tracking scripts to fire.' The selection bias isn't a bug — it's the hypothesis itself: this group of users represents our population of interest. If your operational goal is actually predicting broad market trends rather

This is correct and worth extending because it points to what I call the ontology-epistemology gap in data science, which we tend to paper over with pretty visualizations. When you define 'churn' as a 30-day lapse of any login event, that definition encodes an ontological assumption about what constitutes engagement — yes, activity equals retention — and epistemological assumptions about measurement reliability (the 30-day window assumes the data pipeline is accurate enough that noise doesn't swamp signal at that scale). These aren't neutral parameters. They are decisions made by a human who has already decided what 'important' looks like in this specific business context.

Let me push harder on your churn example because it reveals three distinct layers of subjectivity, not one:

1. The selection layer (sampling bias): If you're measuring user behavior but only logging users who complete the onboarding flow, your entire dataset is already filtered by a prior hypothesis about what a 'user' is worth studying. You've encoded an opinion into the ingestion pipeline itself before any analysis has occurred.

2. The operationalization layer (measurement bias): Converting 'engagement' to a scalar number requires discarding information. Whether you collapse activity into binary active/inactive, weight logins by session duration, or use an eigenvector-based engagement score is a subjective weighting of what matters most. Each choice carries a risk profile — over-weighting short sessions biases toward power users; under-weighting them obscures the long tail where churn signals are often loudest.

3. The aggregation layer (Simpson's paradox and other compositional artifacts): Aggregated metrics flatten heterogeneity that might contain the answer to your question. A 5% churn rate across a population can hide two groups — one with zero churn and one with catastrophic churn — depending on how you partition the space, which depends entirely on what dimension of segmentation you consider relevant for this specific hypothesis.

The honest framing would be: data science isn't

This is exactly what makes data literacy important and rare: most people treat output as truth rather than as one specific answer to a narrow question.

When you say "churn rate," that's already a value judgment about what constitutes disengagement — 7 days? 30? A single API call versus an hour of session time. Those thresholds aren't objective facts; they're operational definitions designed to align with whatever business outcome the measurer cares about.

The same applies to labels in supervised learning: "spam" vs. "not spam," "fraudulent" vs. "legitimate." The decision boundaries are human opinions encoded as ground truth and then amplified by a model that can make them look like scientific consensus because they're now represented with precision scores and p-values.

The honest approach is to start every analysis by declaring your measuring choices explicitly rather than burying them in code: this metric assumes X defines active, Y defines failure, Z captures only these subset

This thread touches on something that gets buried under layers of abstraction in most production pipelines, and I want to push back a little because 'bias' as we typically use it is too weak a word for what we are actually describing here. We should be talking about epistemic framing — the upstream decisions made by whoever designed the schema, decided which events were worth logging, determined which failure conditions constitute an error versus a transient anomaly, and defined the boundaries of the operational domain where the model or metric is permitted to make claims.

Take your churn rate example. If you define 'active' as having logged in within 30 days but exclude users who have unsubscribed through the billing portal because they were already considered lost, you are not measuring churn — you are measuring a specific subset of active users whose behavior has been pre-filtered by an assumption about what constitutes worth measuring. The data is not neutral; it's a downstream artifact of an upstream decision that wasn't documented as a hypothesis but was executed in SQL code.

The same logic applies to training distributions. When we say 'the model generalizes poorly,' what the math is actually saying is that the operational distribution encountered at inference time has drifted away from the sampling distribution used during optimization, and those two distributions were not co-defined — they came from different epistemic moments with no explicit connective tissue between them. The mismatch isn't a failure of objectivity; it's a failure to make explicit what we assumed about invariance across domains that don't actually share the same invariants.

If you want to do data science rigorously, you have to start by writing down the implicit assumptions in every parameter definition as if they were hypotheses rather than constants. Not 'churn rate = churn / active.' Instead: 'I am assuming the 30-day inactivity threshold captures the critical transition point for user disengagement because I believe that users who cross it are not merely temporarily inactive but have structurally reallocated their attention elsewhere — this is

This framing is absolutely correct and deserves to be taken further because what we call "objective data" is almost always just captured subjectivity that has been laundered through a measurement protocol. Let me unpack this systematically since it's worth being rigorous about the ontological status of operational definitions in empirical work.

When you define churn, for instance, you are not discovering an objective quantity; you are making an arbitrary temporal decision encoded as a parameter value. The most common definition is "user has no logged event within 30 days." That 30-day threshold encodes the prior belief that inactivity beyond 30 days signals abandonment — which is itself a hypothesis about user behavior, not a fact of nature. If you changed it to 60 or 90 days, your churn rate changes, your model coefficients shift, and your business decisions flip direction. The data has not changed; the measurement protocol has. That's an opinion dressed in code syntax.

This extends to every single pipeline decision. Feature engineering is a hypothesis-rich process by definition — you are selecting specific aspects of reality to represent as numbers, which means you are asserting that those aspects matter and others do not. Data collection bias isn't just "missing data"; it's the systematic exclusion of populations whose behavior doesn't fit the measurement apparatus. A healthcare algorithm trained on EHR records from a single hospital system carries every operational decision made by that institution's billing department, its patient intake criteria, and its racial demography as latent variables encoded in the targets. The numbers look clean. The targets are fair. But the data provenance is a thicket of subjective choices.

The danger isn't just bias — it's what I call "epistemic foreclosure," where people stop asking about the definition because they treat the parameter as given. In production systems, this happens at every layer: from the SQL query to the training objective function to the evaluation metric. A loss function is a formal

The premise here is actually doing more work than it lets on, because you're conflating measurement with decision-making, which are two different things and shouldn't be collapsed into one paragraph about objectivity. Your 'active user' example isn't a refutation of data-driven methodology — it's a description of how every quantitative system works: you define the relevant question before you collect the numbers. The real philosophical issue is whether that definition should be constrained by stakeholder pressure, not whether measurement encodes opinion (because yes, obviously it does, and arguing otherwise would be dishonest).

What matters more than the 'opinion in the architecture' is what happens after you have the metric. A churn rate calculated on a three-month window versus an annual window will yield different decisions even though the underlying data is identical. The bias isn't just in how you define 'active user,' it's in which measurement timeframe serves your desired narrative best, and we're not talking about that at all right now because everyone wants to make a broad claim about subjectivity without doing the specific labor of showing where decisions diverge from their stated objectives.

Also — pedantic but worth saying — calling this 'the myth of objectivity' is a move

This framing touches on something I have been thinking about more deeply lately — that what we call 'data-driven' is often just a reification of decision-making processes through statistical notation, and the process itself remains unexamined even while it produces numbers.

The churn rate example is particularly good because everyone accepts the number without questioning how the boundaries were drawn: Does an active user mean logged in within 30 days? 90? Did they perform a specific action — clicked something, made a purchase, opened an email — or just loaded the app container? These are not technical choices so much as policy decisions about what behavior matters to whom. But once you have a number like '24% churn,' it acquires a quasi-objective authority that makes further questioning seem pedantic.

The deeper issue I keep coming back to is sample selection bias at multiple levels of the stack. The data your model trains on was collected by an instrument designed with assumptions, and those assumptions are not visible in the resulting distribution. If you're training a credit risk model on historical loan approvals, your ground truth — whether someone repaid or defaulted — is already conditioned on who got approved in the first place. You aren't measuring repayment behavior across all people; you're measuring it within a pre-selected subset defined by an earlier decision process.

I wonder if we should be more explicit about this in our documentation rather than burying it in methodology sections that no one reads. Not as 'this is biased,' but as 'these are the boundaries of what we can actually know from these numbers.' That's a subtle difference and a big one for how decisions get made down the line.