Problem Statement

Given a NumPy array, rescale its values onto a common range or distribution — and do it in a way that still works if one value in the data is unusually large or small.

Example

Input:

A small salary dataset — five ordinary salaries, and one executive salary that's wildly larger than the rest:

[42000, 45000, 47000, 44000, 46000, 250000]

That last value isn't a typo. It's the realistic case — the point where naive normalization quietly stops working.

Using Min-Max Normalization

Min-max rescales everything into a fixed [0, 1] range, based on the minimum and maximum values in the data:

import numpy as np

data = np.array([42000, 45000, 47000, 44000, 46000, 250000])

min_max = (data - data.min()) / (data.max() - data.min())
print(np.round(min_max, 3))

[0.000 0.014 0.024 0.01  0.019 1.000]

Look at what happened to the five ordinary salaries: they're squeezed into a range from 0.000 to 0.024 — barely distinguishable from each other — while the one outlier claims the entire rest of the scale by itself. Min-max uses only two data points, the minimum and the maximum, to define the whole scale. One extreme value, and the other 83% of your data (five out of six points) gets compressed into a sliver near zero.

Using Z-Score Standardization

Z-score rescales based on the mean and standard deviation instead — how many standard deviations each value sits from the average:

z_score = (data - data.mean()) / data.std()
print(np.round(z_score, 3))

[-0.484 -0.445 -0.418 -0.458 -0.431  2.236]

This looks like it fixed the problem — the ordinary salaries aren't all clustered at exactly the same spot anymore. But look closer: they still only span a range of about 0.066 (from -0.484 to -0.418), while the outlier sits all the way out at 2.236. Here's the mechanism, made concrete: the mean of all six points is 79,000 — but the mean of just the five ordinary salaries, without the outlier, is 44,800. One data point pulled the average up by more than 76%. Standard deviation is calculated from how far every point sits from that inflated mean, so it gets dragged upward too. Z-score is less visually dramatic about it than min-max, but it's not actually immune to the same problem.

Note: data.std() defaults to population standard deviation (ddof=0) in NumPy, while scikit-learn's StandardScaler uses sample standard deviation (ddof=1) by default. The two will give you slightly different numbers on the same data — that's not a bug in either one, just a different default assumption about whether your array is the whole population or a sample of it.

Using Robust Scaling (Median and IQR)

Robust scaling swaps out both the mean and the standard deviation for statistics that don't move much when there's an outlier in the data — the median, and the interquartile range (the spread of the middle 50% of the data). The result is a scale where the median lands at 0 and the IQR itself becomes the unit of measurement — a value of 1 means "one IQR above the median," regardless of how extreme the raw numbers were:

median = np.median(data)
q1, q3 = np.percentile(data, [25, 75])
iqr = q3 - q1

robust = (data - median) / iqr
print(np.round(robust, 3))

[-1.4 -0.2  0.6 -0.6  0.2  81.8]

This is the difference that actually matters. The five ordinary salaries now span a real, usable range — -1.4 to 0.6 — instead of being crushed into a sliver near zero or near each other. The median (45,500) and IQR (2,500) are calculated from the middle of the data, so one extreme value barely moves them — compare that 2,500 IQR to the mean's jump from 44,800 to 79,000 in the z-score section above. And the outlier itself, instead of quietly dominating the scale, gets a score of 81.8 — a number that immediately flags it as extreme rather than hiding it among values that look almost the same.

Small footnote: np.percentile's default interpolation method for computing Q1/Q3 can differ slightly from how scikit-learn's RobustScaler computes them internally. On this dataset the numbers line up cleanly, but on messier or larger datasets, don't be surprised by small discrepancies between the two if you're comparing hand-rolled NumPy against the scikit-learn equivalent.

Watch Out For

• Constant or near-constant arrays. Min-max divides by (max - min), which is 0 if every value is identical — that's a 0/0 division, producing NaN plus a RuntimeWarning. Z-score hits the same wall if std() is 0. Check that your denominator isn't zero before scaling, or add a small epsilon.

• Single-element arrays. Same problem — a one-element array has a range of 0 and a standard deviation of 0, so both min-max and z-score produce NaN on it.

• Heavily duplicated or zero-inflated data. If the middle 50% of your data is all identical values, IQR can also hit 0, and robust scaling divides by zero the same way the other two do.

• Empty arrays. All three raw NumPy formulas above return an empty array cleanly with no error. Scikit-learn's scalers don't — they raise on empty input. Worth knowing if you're moving between hand-rolled code and scikit-learn in the same pipeline.

• In all of these zero-denominator cases, scikit-learn's scalers handle it gracefully and output 0 instead of crashing — raw NumPy will not do that for you automatically.

Which One Should You Use?

• Data has no serious outliers, and you need values in a fixed range (like [0, 1] for a neural network input)? Use min-max.

• Data roughly follows a normal distribution, no major outliers, and the algorithm you're feeding it assumes standardized features? Use z-score.

• Data has outliers you can't or shouldn't remove, and you still need the typical values to be usefully spread out? Use robust scaling.

• Downstream system requires non-negative or strictly bounded input (certain GLM link functions, image pixel values)? Only min-max guarantees a bounded [0, 1] range — z-score and robust scaling are both unbounded and can go negative.

Before you ship this to production:

scikit-learn ships all three as MinMaxScaler, StandardScaler, and RobustScaler — reach for those directly in a real project instead of hand-rolling the formulas, since they also handle multi-column data and reuse fitted parameters consistently.

If you're scaling training and test data separately, fit the scaler on the training set only, then apply those same fitted parameters to the test set:

scaler.fit_transform(X_train)   # learn parameters from training data only
scaler.transform(X_test)        # apply those same parameters to test data

Fitting a fresh scaler on the test set — or on the combined data — leaks information about the test set into your training process, which quietly inflates how good your model looks during evaluation.

Conclusion

All three scalers agree on the ordinary salaries when there's no outlier in the mix — the differences here only show up because one point in the data is extreme. Min-max and z-score both let that one point dictate the scale for everything else, just through different mechanisms (range vs. mean/std). Robust scaling is the one built specifically to resist that — not because it's always the "better" choice, but because it's the only one of the three that keeps working the way you'd expect once your data stops being clean.

Before Any Method: Ask Why It's Missing

The method you reach for should depend less on which pandas function looks right and more on why the value isn't there in the first place. Roughly, missing data falls into a few buckets: it might be missing for no real reason at all (random gaps — a sensor glitch), missing in a way that's predictable from other columns (a field that's only ever filled in for certain categories), or missing in a way that's actually informative — the fact that it's blank tells you something. A column that's empty in every row could mean "nobody bothered to fill this in" (drop it), or it could mean "this field only applies to some rows and none of them are in this sample" (a sign you need more data, not a deletion). Keep that question in mind as you go — the DataFrame below has four columns, and none of them are missing data for the same reason.

Problem Statement

Given a DataFrame with missing values scattered across different columns, identify them and decide how to handle each one appropriately — not just make the NaNs disappear.

Example

Input:

   day  temp  humidity condition notes
0  Mon    30        65     Sunny   NaN
1  Tue   NaN        68     Sunny   NaN
2  Wed    32       NaN       NaN   NaN
3  Thu    33        70     Rainy   NaN
4  Fri   NaN        72     Rainy   NaN
5  Sat    29       NaN   Cloudy   NaN
6  Sun    28        75     Sunny   NaN

Four columns, four different missing-data situations: temp and humidity have scattered gaps in an otherwise trending numeric sequence, condition is missing one categorical value, and notes is missing every single value. Those aren't the same problem, and they don't have the same fix.

Step 1: Find Out What's Actually Missing

Before fixing anything, see the scope of the problem:

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "day": ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"],
    "temp": [30, np.nan, 32, 33, np.nan, 29, 28],
    "humidity": [65, 68, np.nan, 70, 72, np.nan, 75],
    "condition": ["Sunny", "Sunny", np.nan, "Rainy", "Rainy", "Cloudy", "Sunny"],
    "notes": [np.nan] * 7
})

print(df.isna().sum())

day          0
temp         2
humidity     2
condition    1
notes        7
dtype: int64

That last line is your evidence, not just a count. notes missing in every row supports the "nobody filled this in" theory rather than the "only applies to some rows" theory — there's no pattern to it, it's uniformly empty. That's what tells you dropping it is reasonable, rather than a guess.

One dtype note: isna() catches NaN, None, and pandas' newer pd.NA (used by nullable dtypes like Int64 and string) all in one call — you don't need to check for each separately.

Step 2: dropna() — Decide Which Drop You Mean

The instinct is to just call dropna() and move on. Here's what that actually does on this data:

print(df.dropna())

Empty DataFrame
Columns: [day, temp, humidity, condition, notes]
Index: []

Every single row is gone. Not because every row has real problems — it's because dropna() with no arguments drops a row if any column has a NaN, and notes being empty in every row means every row qualifies. One useless column just cost you your entire dataset.

The fix isn't to avoid dropna() — it's to be specific about which drop you actually mean. First, drop columns that are empty across the board:

df = df.dropna(axis=1, how="all")
print(df)

   day  temp  humidity condition
0  Mon    30        65     Sunny
1  Tue   NaN        68     Sunny
2  Wed    32       NaN       NaN
3  Thu    33        70     Rainy
4  Fri   NaN        72     Rainy
5  Sat    29       NaN   Cloudy
6  Sun    28        75     Sunny

notes is gone, and none of the real data was touched. From here you have finer-grained options than the all-or-nothing default. subset restricts the check to specific columns — drop a row only if a genuinely essential field is missing:

print(df.dropna(subset=["temp"]))

thresh is less aggressive still — keep a row as long as it has at least that many non-null values, regardless of which columns they're in:

print(df.dropna(thresh=3))

   day  temp  humidity condition
0  Mon    30        65     Sunny
1  Tue   NaN        68     Sunny
3  Thu    33        70     Rainy
4  Fri   NaN        72     Rainy
5  Sat    29       NaN   Cloudy
6  Sun    28        75     Sunny

Only Wednesday drops out here — it's the one row missing two values (humidity and condition), so it's the only one that fails the "at least 3 out of 4 non-null" bar.

Step 3: fillna() — for Values Where a Defensible Default Exists

condition is missing one value (Wednesday). It's categorical, so there's no "average" to fall back on — but since weather tends to persist day to day, carrying the previous day's value forward is a reasonable default:

df["condition"] = df["condition"].ffill()
print(df["condition"])

0     Sunny
1     Sunny
2     Sunny
3     Rainy
4     Rainy
5    Cloudy
6     Sunny
Name: condition, dtype: object

Wednesday inherits Tuesday's "Sunny" — not necessarily correct, but a reasonable guess for this kind of data, and clearly flagged as an assumption if you keep a note of what was filled. (Note: fillna(method="ffill") still works on older pandas but is deprecated as of pandas 2.1 — use .ffill() directly, same for .bfill() if you need to fill backward instead of forward.)

Two things worth knowing before you rely on this: .ffill() propagates the last valid value forward indefinitely until it hits a real one — a long run of consecutive NaNs, or a gap right at the start of the column with nothing before it to carry forward, will silently repeat (or fail to fill) far more than you'd expect. .bfill() is the mirror image — it carries the next valid value backward instead — worth reaching for when a gap sits at the very start of your data with nothing earlier to pull from.

Pitfall worth knowing: calling df.fillna(df.mean()) across an entire DataFrame — instead of one column at a time — will either error or skip non-numeric columns without telling you, because you can't average "Sunny" and "Rainy". If you want a blanket fill across only the numeric columns, be explicit about it: df.select_dtypes(include="number").fillna(df.mean()).

Step 4: interpolate() — for Ordered, Trending Numeric Data

temp and humidity are different from condition: they're numeric, ordered by day, and trending — which means a value between the two surrounding known points is usually more accurate than picking one flat number for both gaps. By default, interpolate() does exactly that — linear interpolation, a straight line drawn between the known value before the gap and the known value after it:

df["temp"] = df["temp"].interpolate()
df["humidity"] = df["humidity"].interpolate()
print(df[["day", "temp", "humidity"]])

   day  temp  humidity
0  Mon  30.0      65.0
1  Tue  31.0      68.0
2  Wed  32.0      69.0
3  Thu  33.0      70.0
4  Fri  31.0      72.0
5  Sat  29.0      73.5
6  Sun  28.0      75.0

Tuesday's temperature lands at 31.0 — you can check this by hand: it's the straight-line average of Monday's 30 and Wednesday's 32. Friday's humidity, 73.5, is the average of Thursday's 72 and Saturday's 75. Compare that to what fillna(df["temp"].mean()) would have done: it would have dropped the same flat average (30.4) into both Tuesday and Friday, ignoring that the readings on either side of each gap tell you something different.

Two things to keep in mind: interpolate() only produces something meaningful when the values have a real numeric order — it returns nonsense on categorical, boolean, or text columns, since there's no "value between Sunny and Rainy." And if you have a long run of consecutive gaps rather than single missing points, unrestricted linear interpolation will draw one long flat-ish line across the whole stretch, which can be misleading — the limit= parameter caps how many consecutive NaNs it's willing to fill, so it doesn't quietly paper over a gap that's really too large to guess at.

A Note on How You Assign the Fix

If you see a SettingWithCopyWarning while doing any of this, it usually means you tried to assign into a filtered view rather than the DataFrame itself — something like df[df["temp"].isna()]["temp"] = 0 instead of using .loc[]. It's unrelated to which missing-data method you picked, but it's exactly the kind of warning this topic tends to surface right after you start filtering and fixing values.

Which One Should You Use?

• Want to know how bad the problem actually is, and why, before touching anything? Start with isna().sum() and ask what the pattern (or lack of one) tells you.
• An entire column is empty, or a row is missing so much it's not usable? Use dropna() — scoped with axis, how, thresh, or subset, not called bare.
• A categorical or non-trending value is missing, and a reasonable default exists? Use fillna() (or .ffill()/.bfill() directly) — with a real, deliberate value, not a blind .mean() across every column.
• Numeric data that's ordered and trending, with gaps in the middle? Use interpolate() — and cap it with limit= if the gaps might be long.

Conclusion

The lesson isn't "avoid dropna()" — the article uses it, deliberately, in Step 2. The lesson is to decide which drop you mean: by column first, then by row, scoped with subset or thresh, instead of calling it with no arguments and hoping. Use fillna() when a specific, defensible default exists. Use interpolate() only when the data is genuinely ordered and trending — for anything else, it's not measuring what you think it's measuring.

The short version, before the detail: an inner merge drops any row that doesn't match on both sides — silently. A left merge keeps every row from one side and can show you which ones didn't match. An outer merge keeps everything from both sides, so you see every gap at once. The rest of this article is really just that three-way distinction, worked through in detail.

Problem Statement

Given two DataFrames that share more than one key column, combine them so rows match only when all key columns agree.

Example

Input:

# df_scores
   student_id term  score
0         101   T1     85
1         101   T2     90
2         102   T1     78
3         103   T1     92

# df_attendance
   student_id term  attendance
0         101   T1     95
1         101   T2     88
2         102   T1     80
3         104   T1     99

Notice the two DataFrames don't fully overlap: student 103 has a score but no attendance record, and student 104 has an attendance record but no score. That mismatch is the actual problem this article is about — not just the merge syntax.

Using merge() with on=[...]

The default case — match rows only where every key column agrees, and drop anything that doesn't have a match on both sides:

import pandas as pd

df_scores = pd.DataFrame({
    "student_id": [101, 101, 102, 103],
    "term": ["T1", "T2", "T1", "T1"],
    "score": [85, 90, 78, 92]
})

df_attendance = pd.DataFrame({
    "student_id": [101, 101, 102, 104],
    "term": ["T1", "T2", "T1", "T1"],
    "attendance": [95, 88, 80, 99]
})

result = pd.merge(df_scores, df_attendance, on=["student_id", "term"])
print(result)

   student_id term  score  attendance
0         101   T1     85          95
1         101   T2     90          88
2         102   T1     78          80

This is an inner merge by default — only (101, T1), (101, T2), and (102, T1) exist in both DataFrames, so those are the only rows that survive. Student 103's score and student 104's attendance both silently disappear. That's fine if missing data should be dropped — dangerous if you didn't realize you were dropping it.

If the key columns have different names on each side — say the second DataFrame uses id and term_code instead of student_id and term — use left_on/right_on instead of on:

pd.merge(df_scores, df_attendance_alt,
         left_on=["student_id", "term"],
         right_on=["id", "term_code"])

Two more things worth knowing before you rely on this:

• A NaN in a key column never matches anything, including another NaN — a row with a missing key is silently treated as having no match, on either side.
• Non-key columns that happen to share a name across both DataFrames (like created_at) get renamed automatically with _x/_y suffixes. Use suffixes=("_scores", "_attendance") if you want to control what they're called instead.

Using how='left' with indicator=True

When you need to keep every row from one side — and see exactly which ones didn't find a match — a left merge with the indicator flag on gives you both:

result = pd.merge(
    df_scores, df_attendance,
    on=["student_id", "term"],
    how="left",
    indicator=True
)
print(result)

   student_id term  score  attendance     _merge
0         101   T1     85        95.0       both
1         101   T2     90        88.0       both
2         102   T1     78        80.0       both
3         103   T1     92         NaN  left_only

Every row from df_scores is kept, whether or not it matched. Student 103 shows up with attendance as NaN and _merge flagged as left_only — pandas is telling you directly, row by row, which side of the merge each row actually came from. Filtering result[result["_merge"] == "left_only"] gives you exactly the rows an inner merge would have thrown away without telling you. (If you want to keep that column for further analysis under a clearer name, pass indicator="match_status" instead of True.)

Common Pitfall: Duplicate Keys

This is the gotcha that causes the most confusing production bugs in multi-column merges, and it's worth seeing once instead of just being warned about. If your key columns aren't actually unique on one side, merge doesn't error — it silently multiplies rows:

df_scores_dup = pd.DataFrame({
    "student_id": [101, 101, 101],
    "term": ["T1", "T1", "T2"],
    "score": [85, 87, 90]
})

result = pd.merge(df_scores_dup, df_attendance, on=["student_id", "term"])
print(result)

   student_id term  score  attendance
0         101   T1     85          95
1         101   T1     87          95
2         101   T2     90          88

Two different scores for the same (101, T1) key both matched the single attendance row — the attendance value 95 now appears twice, once per duplicate. In a small example it's obvious. On a real dataset with thousands of rows, this is exactly how a merge quietly inflates your row count and nobody notices until a downstream aggregate is mysteriously too high.

The fix is validate, which turns this from a silent bug into an explicit error:

pd.merge(df_scores_dup, df_attendance, on=["student_id", "term"], validate="1:1")

MergeError: Merge keys are not unique in left dataset; not a one-to-one merge

Use validate="1:1" when you expect exactly one match per key pair on each side, or "1:m"/"m:1" if one side is genuinely expected to repeat. It costs nothing to add and catches this class of bug before it reaches production.

Using how='outer' to Reconcile Both Sides

When neither DataFrame should lose rows — you want the full picture, gaps and all — an outer merge keeps everything from both sides:

result = pd.merge(df_scores, df_attendance, on=["student_id", "term"], how="outer")
print(result)

   student_id term  score  attendance
0         101   T1   85.0        95.0
1         101   T2   90.0        88.0
2         102   T1   78.0        80.0
3         103   T1   92.0         NaN
4         104   T1    NaN        99.0

Now both mismatches show up: student 103 with no attendance, student 104 with no score. Neither an inner merge nor a plain left merge would have shown you both at once.

Notice score and attendance became floats (85.0 instead of 85) — pandas can't represent a missing value in an integer column, so introducing any NaN forces the whole column to float. This is worth knowing before it surprises you downstream in code that expects integers.

From here, don't reach for fillna(0) without thinking about what 0 actually means for your data — a missing attendance record isn't the same fact as "attended zero classes," and filling it with 0 quietly turns "we don't know" into a false claim. If a specific default genuinely makes sense for your case, say so explicitly and comment why:

# 0 here means "no record exists yet," not "confirmed zero attendance" —
# only fill this way if your downstream logic treats them the same.
result["attendance"] = result["attendance"].fillna(0)
result["score"] = result["score"].fillna(0)

# now safe to cast back to int, since there are no NaNs left
result[["score", "attendance"]] = result[["score", "attendance"]].astype(int)

Often the better move is to leave the NaNs as-is and handle the two mismatched rows deliberately — a separate lookup, a flag for manual review, whatever fits — rather than picking a placeholder number that might get misread as real data later.

Performance note: on large DataFrames, multi-column merges are one of the more expensive pandas operations. If you're merging millions of rows, setting sort=False (the default since recent pandas versions) and making sure both DataFrames are already sorted on the key columns beforehand can meaningfully speed things up.

Which One Should You Use?

• Only care about rows that exist in both DataFrames? Use the default merge(on=[...]) — inner join.
• Need to keep every row from one side and see what didn't match? Use how='left' with indicator=True.
• Need to see and resolve mismatches from both sides? Use how='outer', then decide how to handle the NaNs explicitly.
• Not sure whether your keys are actually unique? Add validate="1:1" before you trust any of the above.

Conclusion

Don't default to the inner merge just because it's the shortest call — default to it only once you've confirmed that dropping unmatched rows is actually what you want. If you haven't confirmed that yet, run how='left' with indicator=True first and see what an inner merge would have silently thrown away. Add validate any time you're not certain your keys are unique — it's one keyword argument standing between you and a duplicated-row bug that's much harder to spot after the fact than before it.

Problem Statement

Given a DataFrame with duplicate rows, remove the duplicates so each unique row appears only once.

Example

Input:

   name    city  age
0  Amit   Delhi   28
1  Neha  Mumbai   25
2  Amit   Delhi   28
3  Riya    Pune   30
4  Neha  Mumbai   25
5  Amit   Delhi   31

Output:

   name    city  age
0  Amit   Delhi   28
1  Neha  Mumbai   25
3  Riya    Pune   30
5  Amit   Delhi   31

Rows 2 and 4 are exact duplicates of rows 0 and 1, so they're dropped. Row 5 looks similar to row 0 — same name, same city — but the age is different, so it's not an exact duplicate and stays.

Using drop_duplicates()

The simplest and most common approach — pandas checks every column by default and drops exact-match rows:

import pandas as pd

df = pd.DataFrame({
    "name": ["Amit", "Neha", "Amit", "Riya", "Neha", "Amit"],
    "city": ["Delhi", "Mumbai", "Delhi", "Pune", "Mumbai", "Delhi"],
    "age":  [28, 25, 28, 30, 25, 31]
})

result = df.drop_duplicates()
print(result)

   name    city  age
0  Amit   Delhi   28
1  Neha  Mumbai   25
3  Riya    Pune   30
5  Amit   Delhi   31

By default, drop_duplicates() keeps the first occurrence of each exact duplicate and drops the rest. You can flip that with keep='last', or drop every copy entirely — originals included — with keep=False:

print(df.drop_duplicates(keep='last'))
print(df.drop_duplicates(keep=False))

   name    city  age
2  Amit   Delhi   28
3  Riya    Pune   30
4  Neha  Mumbai   25
5  Amit   Delhi   31

   name  city  age
3  Riya  Pune   30
5  Amit Delhi   31

You can also limit the check to specific columns with subset, which treats rows as duplicates if they match on just those columns — even if other columns differ:

print(df.drop_duplicates(subset=["name", "city"]))

   name    city  age
0  Amit   Delhi   28
1  Neha  Mumbai   25
3  Riya    Pune   30

This is a different result from the default call above. Checking only name and city, row 5 now counts as a duplicate of row 0 — even though their age differs — so it's dropped this time. Whether that's what you want depends on whether age should matter to the comparison.

Pro tip: dropping duplicates leaves gaps in the index (0, 1, 3, 5 instead of 0, 1, 2, 3). If that trips up code downstream that assumes a clean sequential index, chain on .reset_index(drop=True).

Using duplicated() as a Boolean Mask

You can get the same effect as drop_duplicates() manually with duplicated(), which is worth knowing when you want the mask itself — to log or inspect which rows are duplicates before removing anything:

mask = df.duplicated()
print(mask)

result = df[~mask]
print(result)

0    False
1    False
2     True
3    False
4     True
5    False
dtype: bool

   name    city  age
0  Amit   Delhi   28
1  Neha  Mumbai   25
3  Riya    Pune   30
5  Amit   Delhi   31

duplicated() returns True for every row that's an exact repeat of an earlier one. Row 5 comes back False — same as with the default drop_duplicates() call — because it isn't an exact duplicate of row 0, just a partial match on name and city. Flipping the mask with ~ and indexing with it gives the same result as drop_duplicates(), but now you've got the mask to work with directly.

Using groupby() to Deduplicate and Aggregate

When "duplicate" rows differ in some columns and you need a deterministic rule for combining them, rather than arbitrarily picking one, groupby() is the right tool:

result = df.groupby(["name", "city"], as_index=False).agg({"age": "max"})
print(result)

   name    city  age
0  Amit   Delhi   31
1  Neha  Mumbai   25
2  Riya    Pune   30

This is the case where the three methods genuinely disagree. drop_duplicates() kept Amit's age as 28 — the first occurrence, arbitrarily. groupby().agg({"age": "max"}) looks at all of Amit's rows (28 and 31) and explicitly keeps the larger value. If 28 were a stale reading and 31 the correct one, the first two methods would have silently kept the wrong number — groupby is the one that actually resolves the conflict instead of ignoring it.

One thing to note: groupby() sorts its output by the grouping keys by default (alphabetically here: Amit, Neha, Riya), while drop_duplicates() preserves the original row order. If you run both on a differently-ordered DataFrame, don't be surprised that the row order doesn't match — that's expected, not a bug.

Which One Should You Use?

• Duplicates are exact matches, and you just want them gone? Use drop_duplicates().
• You need to know exactly which rows got dropped before committing to it? Use duplicated() to get the mask first.
• "Duplicate" rows actually differ in some columns, and you need a rule for combining them? Use groupby().agg().

Conclusion

Start with drop_duplicates() — it covers the common case in one line. Switch to duplicated() when you need visibility into what's being removed before you commit to it. Switch to groupby().agg() the moment "duplicate" stops meaning "identical" and starts meaning "needs a rule" — that's the case the other two methods can't actually handle correctly.

An internal, unreleased AI system had cracked part of the Navier-Stokes existence and smoothness problem. One of the seven Clay Mathematics Institute Millennium Prize Problems. A $1 million reward. Untouched for most of a century.

The internet did what the internet does. It split into two arguments.

One argument was about credit. NYU mathematician Tristan Buckmaster and his collaborator Levent Alpöge — a mathematician who works at Anthropic, but who was pursuing this work independently of his employer — said they'd been developing closely related unpublished work. Buckmaster publicly questioned whether OpenAI's system got there on its own.

The other argument barely happened at all. It's the actual mathematics. What the problem asks. What mechanism the proof proposes. How a swarm of AI agents allegedly built it. What "verified in Lean" does and doesn't mean.

That's the argument this piece is having.

The credit dispute is real. It's worth knowing about. It's covered in depth elsewhere. We're not relitigating it here.

Quick timeline, so the hour-counts below actually mean something:

• Aug 28 — OpenAI starts training the new internal model behind this result
• Sept 1 — After hearing rumors that two Millennium Prize problems had fallen, OpenAI points the model at all of them
• Sept 5 — Agents reach the Navier-Stokes resolution, ~88 hours after launch
• Sept 6 — Lean formalization and verification wraps, ~17 hours later
• Sept 8 — Public announcement

One more thing before we start. OpenAI hasn't released its internal model, its agent orchestration system, or the full 166-page proof as executable code. The Python snippets below are mine. Simplified. Built to make one idea concrete at a time. None of them reproduce or verify OpenAI's actual proof — the real formal proof, in Lean 4, is public on OpenAI's GitHub. Each snippet is labeled with exactly what job it's doing: building intuition, making a quantitative point, or illustrating structure. Nothing more.

What the Problem Actually Asks

Navier-Stokes describes how fluids move. Air over a wing. Blood through a vessel. Water in a pipe.

The question underneath the Millennium Prize is deceptively simple: start with smooth, finite-energy, divergence-free initial data in three dimensions. Does the fluid stay smooth forever? Or can it blow up — velocity going to infinity at some point — in finite time?

The Clay Institute actually allows four ways to answer this. Labeled (A) through (D).

(A) and (B) would prove global smoothness always holds — for whole space, and for the periodic case.

(C) and (D) would disprove it. You'd need a smooth, divergence-free initial condition and a smooth external force where the solution blows up in finite time. Again, whole space and periodic torus.

OpenAI claims (C) and (D). A disproof. Not "fluids always stay smooth" — the opposite.

The Hard Constraint Nobody Mentions

Here's why this resisted proof for decades.

You can trivially blow up a fluid by applying an infinitely large or singular external force. That doesn't count. Nobody's impressed by that.

The real ask: velocity diverges to infinity in finite time, while the applied force stays smooth and finite, while total kinetic energy stays bounded the entire time, while viscosity — the fluid's internal friction, the thing that normally damps everything out — stays strictly positive throughout.

Infinite velocity. Bounded energy. Smooth forcing. All at once. For the entire evolving 3D field.

That's the crux of the whole problem.

A number that makes the constraint real

A self-similar blow-up typically has velocity scaling like v(t) ~ (T* − t)−α as time t approaches the blow-up time T*. On its own, that's easy — velocity clearly diverges. The hard part is total kinetic energy, which depends on both velocity and the spatial scale of the motion, staying bounded the whole time. The length scale has to shrink at exactly the right rate to cancel the growing velocity in the energy calculation.

This snippet is a numerical illustration of that balance. Not a proof. Just a demonstration that the balancing act is at least numerically achievable in a toy setting.

import numpy as np

T_star = 1.0   # blow-up time (illustrative)
alpha = 0.5    # self-similar velocity scaling exponent (illustrative, not OpenAI's real exponent)
times = np.array([0, 0.5, 0.8, 0.9, 0.95, 0.99, 0.999, 0.9999])

def velocity_scale(t, T_star, alpha):
    return (T_star - t) ** (-alpha)

# Choose the length-scale exponent so velocity^2 * length_scale^3 (toy "energy") stays constant.
# This is the balancing act a real blow-up construction must achieve rigorously, everywhere, not just at sample points.
length_power = 2 * alpha / 3

for t in times:
    v = velocity_scale(t, T_star, alpha)
    length_scale = (T_star - t) ** length_power
    toy_energy = v**2 * length_scale**3
    print(f"t={t:.4f}  velocity={v:9.3f}  length_scale={length_scale:8.5f}  toy_energy={toy_energy:6.4f}")

t=0.0000  velocity=    1.000  length_scale=1.00000  toy_energy=1.0000
t=0.5000  velocity=    1.414  length_scale=0.79370  toy_energy=1.0000
t=0.8000  velocity=    2.236  length_scale=0.58480  toy_energy=1.0000
t=0.9000  velocity=    3.162  length_scale=0.46416  toy_energy=1.0000
t=0.9500  velocity=    4.472  length_scale=0.36840  toy_energy=1.0000
t=0.9900  velocity=   10.000  length_scale=0.21544  toy_energy=1.0000
t=0.9990  velocity=   31.623  length_scale=0.10000  toy_energy=1.0000
t=0.9999  velocity=  100.000  length_scale=0.04642  toy_energy=1.0000

Look at that last column. Velocity climbs from 1 to 100. Toy energy sits flat at 1.0 the entire time.

That's the single most important number in this article. You're not just proving something blows up. You're proving it blows up cleanly — every quantity that's supposed to stay finite actually stays finite, right up to the last instant. Doing that rigorously, across an entire evolving 3D vector field, continuously, not at eight sample points — that's the multi-decade difficulty. The blow-up isn't the hard part. Controlling it this precisely is.

The Proposed Mechanism: A Vortex That Feeds Itself

Picture a vortex. A rotating column of fluid. Now picture it spiraling inward while stretching along its own axis — like twisting and pulling a piece of rope at the same time.

As the core shrinks, rotational velocity near the core increases. Here's the part that matters: this isn't a one-time effect. It's a loop. Faster rotation stretches the vortex further. That shrinks the core further. Which speeds up rotation further. It compounds until velocity diverges at a finite time T* — even with viscosity still acting on the fluid the whole time.

This style of construction — self-similar blow-up, where the solution looks the same at every scale as it approaches the singularity, just rescaled — is the same class of technique Buckmaster's prior published work explored for related equations. Reportedly the same class he and Alpöge were pursuing in their unpublished work, too.

Watching the loop run

A before/after snapshot could be mistaken for ordinary angular-momentum conservation, which needs no feedback at all. So this version steps through it: current rotation drives further core-thinning, and the thinner core drives faster rotation on the next step.

def simulate_stretching_feedback(steps, core_radius_init, gamma_init, stretch_rate):
    """
    Toy iterative model of vortex self-stretching feedback:
    - angular_velocity depends on circulation / core_radius^2
    - thinning (how fast the core shrinks) is driven by the CURRENT angular_velocity
    - the loop: faster rotation -> faster thinning -> even faster rotation next step
    This is the feedback structure, not just a single shrink-then-measure snapshot.
    """
    core_radius = core_radius_init
    gamma = gamma_init
    history = []
    for step in range(steps):
        if core_radius < 1e-4:
            break
        angular_velocity = gamma / core_radius**2
        peak_velocity = gamma / core_radius
        history.append((step, core_radius, angular_velocity, peak_velocity))
        thinning = stretch_rate * angular_velocity * core_radius
        core_radius = max(core_radius - thinning, 0.0)
    return history

history = simulate_stretching_feedback(steps=10, core_radius_init=0.5, gamma_init=1.0, stretch_rate=0.05)
for step, r, omega, v in history:
    print(f"step={step:2d}  core_radius={r:8.5f}  angular_velocity={omega:10.3f}  peak_velocity={v:10.3f}")

step= 0  core_radius= 0.50000  angular_velocity=     4.000  peak_velocity=     2.000
step= 1  core_radius= 0.40000  angular_velocity=     6.250  peak_velocity=     2.500
step= 2  core_radius= 0.27500  angular_velocity=    13.223  peak_velocity=     3.636
step= 3  core_radius= 0.09318  angular_velocity=   115.170  peak_velocity=    10.732

Watch step 0 to step 3. Core radius drops from 0.5 to 0.093. Angular velocity jumps from 4 to 115 — almost 29x, in three steps.

That's the feedback signature. It's not that a smaller core happens to spin faster — that's true of any static angular-momentum quantity, feedback or not. It's that rotation actively drives further thinning, which drives faster rotation, and the two amplify each other until the toy model's own step size can't resolve it anymore. A real proof has to control that compounding continuously, for all time up to T* — which is exactly why it needs the energy bound from the section above to hold the whole way through.

How the Multi-Agent Pipeline Reportedly Worked

This wasn't one long, continuous search. OpenAI's own description breaks it into stages.

Decompose first. Instead of pointing every agent at the full 3D viscous problem immediately, OpenAI split it into variants of increasing difficulty.

Warm up on something easier. Nearly 100 agents went at the Euler equations blow-up problem — Euler is Navier-Stokes without viscosity, mathematically simpler. Solved in roughly 50 hours.

Transfer the insight. Techniques and partial results from that warm-up got fed back in as scaffolding for the harder, viscous case.

Cross-pollinate with Codex. OpenAI used its Codex coding agent as a consolidation layer — merging the strongest ideas surfacing across different agent groups, instead of leaving thousands of them working in total isolation.

Run it at scale. Across the full campaign, roughly 10,000 agents coordinated over 88 hours, using an internal next-generation model reported to be more capable than the publicly available GPT-6 Astra. That's what built the full analytical proof.

Verify it independently. Once the proof existed in natural language and mathematics, it still had to go into Lean 4 and get mechanically checked. A separate 17-hour process, using GPT-6 Astra.

Now — a caveat worth stating plainly. "10,000 agents" almost certainly doesn't mean 10,000 independent reasoning engines, each chasing a distinct proof strategy from scratch. At this scale, a swarm typically means a large share of sub-tasks, validation passes, retries on dead branches, and specialized roles — drafting, checking, formalizing — rather than 10,000 parallel mathematicians. OpenAI hasn't published the breakdown. Treat the headline number as an upper bound on parallelism, not a literal head count.

And "brute force" here doesn't mean exhaustive search. Agents weren't randomly trying algebraic manipulations until something stuck — that's computationally hopeless at this depth. It was massive parallel exploration on human-decomposed sub-problems, with an automated step merging progress across groups.

Where the Credit Dispute Touches the Method

Separate question from how the pipeline worked: what fed into it.

Buckmaster has said he doesn't know whether OpenAI's system was influenced by his and Alpöge's unpublished work — possibly through their use of OpenAI's Codex tool in the weeks before the announcement. OpenAI has denied using their proof or prompts to direct its agents. Neither claim is independently settled as of this week. It's a separate question from whether the pipeline above is technically sound. We're not speculating further on it here.

What Lean Verification Actually Checks

"Verified in Lean" is a narrower claim than "mathematicians confirmed this solves the Millennium Prize problem." The gap matters.

Lean 4 is a formal proof assistant built on dependent type theory. It doesn't read a natural-language argument and vouch that it "sounds right." Every definition, lemma, and inference gets re-expressed in Lean's own formal language. A small, trusted kernel mechanically checks that each step's premises are fully discharged against everything proven before it. Zero tolerance for gaps. If the formal statement faithfully represents what it claims to, a successful Lean check is strong evidence the internal logic is airtight.

What it does not check: whether the formal statement is a faithful translation of the Clay Institute's exact problem. That still needs expert human mathematicians. That review wasn't complete as of this week.

What's Still Open

Three things, stated plainly.

Independent peer review is incomplete. Lean confirms internal logical consistency. It doesn't by itself confirm the formal theorem is an accurate encoding of the Millennium Prize problem's exact conditions.

The provenance question is unresolved. Whether OpenAI's agents were influenced, directly or indirectly, by Buckmaster and Alpöge's unpublished work is disputed by both sides. Not independently settled.

The pipeline isn't independently reproducible yet. The internal model and agent framework are unreleased. Outside researchers can't verify the "10,000 agents, 88 hours" claim firsthand.

Bottom Line

Here's the one technical claim worth holding onto: a specific self-stretching vortex construction is claimed to produce a mathematically valid finite-time blow-up for Navier-Stokes, under the simultaneous constraints of bounded energy and smooth, finite forcing. A balance that resisted rigorous construction for the better part of a century.

Whether a decomposed swarm of AI agents assembled that construction through genuine mathematical insight, or converged on a path other researchers had already mapped — that's a separate, unresolved question.

Only the broader mathematical community's ongoing review will settle it.

An index is like the index at the back of a book. Or a phone book, sorted by last name. Look up "Smith," flip to the right page, done. No index? You'd read the whole book cover to cover, hunting for one name.

Simple. Clean. Easy to remember.

It's also true.

Which is exactly why it's easy to stop there.

Because that analogy is where every single tutorial on database indexes stops. Not because it's lazy — it genuinely earns its keep for a first exposure. But it hands you a comfortable picture and walks away, right before the part that actually matters. It tells you that a sorted lookup beats scanning everything. It never tells you what that sorted thing actually is, how it gets built, or what it costs you every time you write a row.

So here's the deal. Strip the book metaphor away completely. Let's look at what an index actually is — a real structure, sitting in memory, built a specific way, with real tradeoffs you're paying whether you know it or not.

And here's the part that makes this worth your next five minutes: that same structure, wearing a different name, is quietly running every AI search feature you've used this year. Same idea. Different disguise. Keep reading and you'll never look at "add an index" the same way again.

The explanation you already have

Here's the query everyone writes at some point:

SELECT * FROM orders WHERE customer_id = 48213;

No index on customer_id? Your database checks every single row. One at a time. Was it 48213? No. Next. Was it 48213? No. Next. On a table with a thousand rows, you won't even notice. On a table with ten million rows, you'll notice. You'll notice for eight full seconds while your users stare at a spinner.

Add an index:

CREATE INDEX idx_customer_id ON orders(customer_id);

Same query. Same data. Now it comes back in milliseconds.

That's the part everyone tells you. Fast lookup, sorted structure, book index, phone book, pick your metaphor. True. Also the ceiling, not the floor. Below that ceiling is where this gets interesting.

What's actually happening in memory

An index isn't a setting you flip. It's not a hint you give the database. It's an entirely separate structure — usually something called a B-tree — that your database builds and maintains, sitting right alongside your actual table: living on disk, and fighting for space in your database's memory cache right along with everything else you're querying.

Picture it like this. A root node at the top. Branch nodes underneath, splitting the data into ranges. Leaf nodes at the bottom, holding the actual sorted values and a pointer back to where the real row lives.

                [ 50 ]
              /        \
        [ 20 ]          [ 80 ]
        /    \           /    \
   [1-19]  [21-49]  [51-79]  [81-∞]

Looking for customer 48213? You don't scan a million rows. You start at the root, ask "bigger or smaller than 50?", drop down one level, ask again, drop down again — a handful of comparisons instead of a million. That's the entire trick. That's why O(n) — check every row — becomes O(log n) — check a few, cut the problem in half each time. On a million-row table, that's the difference between roughly a million comparisons and about twenty.

Twenty. Not twenty thousand. Twenty.

Here's what EXPLAIN ANALYZE shows you before the index exists:

Seq Scan on orders  (cost=0.00..21847.00 rows=1 width=96) (actual time=0.021..187.442 rows=1 loops=1)
  Filter: (customer_id = 48213)
  Rows Removed by Filter: 999999
Execution Time: 187.501 ms

Read that middle line again. Rows Removed by Filter: 999999. Your database checked 999,999 rows it didn't need, just to find the one it did.

Now with the index:

Index Scan using idx_customer_id on orders  (cost=0.43..8.45 rows=1 width=96) (actual time=0.018..0.019 rows=1 loops=1)
  Index Cond: (customer_id = 48213)
Execution Time: 0.031 ms

One row checked. Not a million. One.

But — and this is the part the book-index analogy conveniently skips — that speed isn't free. Every time you INSERT, UPDATE, or DELETE a row, the database doesn't just touch your table. It has to go find the right spot in that B-tree and update it too — and a real B-tree stays balanced, which means an insert can trigger a node split, cascading a rebalance up the tree just to keep every lookup path the same length. More indexes, more of that work, every single write. You're not getting fast reads for nothing. You're trading write speed for read speed, on purpose, every single time you type CREATE INDEX.

That's not a footnote. That's the whole deal — and it's also the whole reason this bites people. It runs fine on your laptop. A few thousand rows, everything's instant, you ship it, you feel good. Then production happens. The table's grown to two million rows, and that same query that felt instant now takes eight seconds. Nobody added the index. The sequential scan that was invisible at a thousand rows becomes very visible at two million. Same code. Different amount of data to plow through.

Why this constrains real systems

Here's the part that surprises people: more indexes isn't always the fix. Production DBA teams spend real time removing indexes, not just adding them. A table with ten indexes pays that rebalancing cost ten times on every single write — that's index bloat, and on a hot table (constant inserts, constant updates) it can slow writes down enough that the "fix" becomes the new bottleneck. An index nobody's queries actually use is pure cost, no benefit, sitting there quietly taxing every write forever. Knowing what an index actually is — not a free setting, but a structure with its own upkeep — is what tells you when to stop adding them.

Now here's where it gets genuinely interesting, and where this stops being "just a SQL tip."

Every AI product you've used this year that does semantic search — ask a question, get back the relevant documents, not just ones with matching keywords — is running on vector databases. And vector databases have their own version of this exact same problem, wearing a different costume.

Instead of comparing customer_id = 48213, they're comparing the "closeness" of two lists of numbers — embeddings — across potentially billions of them. Brute-force compare a query against every single embedding, and you're back to the sequential scan, just with worse math. So instead, most of them use something called HNSW — Hierarchical Navigable Small World graphs — a structure built ahead of time so most of the data can be skipped instead of touched.

Worth being precise here: a B-tree is exact and deterministic, and it's happy living mostly on disk. HNSW is probabilistic — it gives you the approximate nearest matches, not guaranteed-exact ones — and it wants to live in RAM, not disk, to hit the speeds vector search is known for. Different mechanics entirely. But strip away the implementation and it's the same shape of tradeoff: pay upfront in memory and build time, so that lookups cost you almost nothing later. And it has the same catch, too — HNSW's speed depends on that graph structure fitting in memory, the same way your hash join from the last article depended on its hash table fitting in RAM. Miss that budget, and the fast thing quietly stops being fast, no error message, no warning, just a system doing more work than you think it's doing.

The code, reframed

Same line you'd write without thinking twice about it:

CREATE INDEX idx_customer_id ON orders(customer_id);
-- Builds a B-tree: a sorted structure of customer_id values,
--   each one holding a pointer back to its actual row.
-- Turns a linear O(n) row-by-row scan into an O(log n) tree walk —
--   a handful of comparisons instead of a million.
-- Costs you on every write: INSERT, UPDATE, and DELETE on this table
--   now also update this structure, every single time.
-- The exact same trade every vector database makes when it builds
--   an HNSW graph for semantic search — memory and build time, spent
--   once, so every future lookup gets to be nearly instant.

Run EXPLAIN ANALYZE on it yourself. Watch Seq Scan turn into Index Scan. Watch Rows Removed by Filter: 999999 disappear entirely. That's not a metaphor anymore. That's your database showing you, in plain text, exactly which of these two things it just did.

Where this leaves you

"Just add an index" was never magic — it's a structure, and a tradeoff, the same bet your database and every AI search feature you use make every time speed matters more than the cost of getting there.

Next time someone tells you to "just add an index" — you'll know exactly what you're building, and exactly what you're paying for it.

SELECT
    e.employee_id,
    e.employee_id,
    e.department
FROM employees e
JOIN sales s
  ON e.employee_id = s.employee_id;

SELECT
    e.employee_id,
    e.employee_name,
    e.department
FROM employees e
JOIN sales s
  USING (employee_id);

It's a fun meme. It also gets the performance story wrong. ON and USING produce the exact same execution plan — same cost, same performance, same everything. USING is shorthand for when both columns share a name, nothing more. The meme isn't a perf tip; it's a style preference dressed up as one.

The real performance story isn't in what you typed. It's in what the database does after you hit enter. A JOIN isn't a keyword — it's an instruction to go find matching rows in memory. How it finds them is the entire reason your query is fast or slow, syntax aside. That's what this article is actually about.

The explanation you've already heard

Here's the query from the meme, cleaned up:

SELECT
    e.employee_id,
    e.employee_name,
    e.department,
    s.net_sales
FROM employees e
JOIN sales s USING (employee_id);

And here's the standard explanation you've gotten for it: "A JOIN combines rows from two tables where a condition matches."

That's true. It's also not wrong. But notice what it describes — the output, not the work. It's the SQL equivalent of describing a for loop as "it repeats" without saying what's being repeated, how many times, or at what cost. That description is a ceiling, not a floor. Everything interesting about JOIN performance lives below it.

So let's go below it.

What's actually happening in memory

When your database sees JOIN, it doesn't run "the JOIN algorithm" — it picks between several, based on table size, indexes, available memory, and one thing that trips people up: table statistics. The optimizer isn't looking at your actual data when it chooses a strategy — it's looking at a cached estimate of that data's shape. If those statistics are stale — say, your database still thinks a table has 10 rows when it actually has 2 million — the optimizer will confidently pick nested loop for a table that's grown far past the point where that's a good idea, regardless of how much RAM you have sitting idle. Same SQL, three completely different things could be happening under the hood, and the optimizer's beliefs about your data are part of why.

Nested loop join — two loops, one memory scan inside another

This is the most literal interpretation of "match rows where a condition holds," and it's exactly what it sounds like:

-- What you write:
SELECT e.employee_name, s.net_sales
FROM employees e
JOIN sales s USING (employee_id);

-- What the engine effectively does:
FOR each row e IN employees:        -- outer loop, scans employees in memory
    FOR each row s IN sales:        -- inner loop, RE-scans sales for EVERY outer row
        IF e.employee_id == s.employee_id:
            emit(e, s)

For every single row in employees, the engine walks the entire sales table looking for a match. That's O(n × m) — multiply the row counts of both tables together, and that's roughly how much work you're asking for. (If the inner table has an index on the join column, this drops to O(n × log m) — the inner loop gets to look the row up instead of scanning for it — but it's still repeating that lookup once per outer row, which is the part that matters here.)

At small scale, this is completely fine — trivial, even. It's usually what your database reaches for automatically when one of the tables is tiny, because the overhead of anything smarter isn't worth it. The problem is what happens when nobody's watching the row counts grow. A join that ran in milliseconds against a 200-row test table can quietly become the slowest thing in your application once that table hits 2 million rows — with the SQL completely unchanged.

Hash join — build once in memory, then probe

This is the one your database usually prefers when tables get bigger, and it's a genuinely different strategy, not just an optimized version of nested loops:

hash_table = {}
FOR each row s IN sales:                 -- BUILD phase: load smaller table into RAM as a hash table
    hash_table[s.employee_id] = s

FOR each row e IN employees:             -- PROBE phase: single pass, O(1) lookup per row
    IF e.employee_id IN hash_table:
        emit(e, hash_table[e.employee_id])

Instead of re-scanning sales for every row of employees, the engine builds a hash table from the smaller table once, then makes a single pass through the larger table, doing a near-instant lookup for each row. That drops the cost to roughly O(n + m) — a massive improvement over nested loops.

Here's the detail that matters most, and the one that's easy to miss if you only think of this as "the fast one": hash joins are RAM-bound, not CPU-bound — until they aren't. The entire hash table has to fit in memory for this to work as described. If it doesn't — because the "smaller" table turns out to be 40GB — the engine doesn't just fail. It spills the hash table to disk in batches and keeps going, silently, with no error and no warning in your application. That spill is the moment the join flips from memory-bound to I/O-bound — from RAM lookups to disk reads and writes — and that flip is usually the actual difference between a query that takes one second and one that takes a full minute. Your query still returns the right answer. It just does it much slower, for a reason that has nothing to do with your SQL and everything to do with how much RAM was available at that moment.

This is why the same JOIN can be fast on your laptop and slow in production. It's not a different query. It's a different amount of memory to work with.

Merge join — both sides pre-sorted, walked in lockstep

The third option only works under a specific condition, but when that condition is met, it's elegant:

-- Requires both inputs sorted on employee_id (via an index or an explicit sort):
e_ptr = 0
s_ptr = 0
WHILE e_ptr < len(employees) AND s_ptr < len(sales):
    IF employees[e_ptr].id == sales[s_ptr].id:
        emit(employees[e_ptr], sales[s_ptr])
        advance both pointers
    ELIF employees[e_ptr].id < sales[s_ptr].id:
        e_ptr += 1
    ELSE:
        s_ptr += 1

If both tables are already sorted on the join column, the engine can walk them side by side with two pointers, advancing whichever one is "behind," and never re-scanning anything. That's O(n + m) with none of the memory pressure of a hash join — no hash table to build, no risk of spilling to disk.

The catch is right there in the setup: both sides have to already be sorted. If they're not, the engine has to sort them first, and that sort cost can erase the entire advantage. Which is exactly what a well-placed index gives you for free — a structure that's already sorted, sitting in memory or on disk, ready for exactly this.

Why this is the whole story, not a technicality

Once you see JOINs this way, a few pieces of common database advice stop being folklore and start being obvious:

"Just add an index" isn't magic — it's a memory-structure claim. An index is a B-tree, sitting in sorted order in memory or on disk. Adding one is what makes a merge join possible without paying a sort cost at query time. That's the actual mechanism behind advice you've probably followed without knowing why it worked.

"It's fast on my machine but slow in production" is usually a hash-join-spilling-to-disk story. Your dev database has a small enough sales table that the hash table fits comfortably in RAM. Production's sales table is 100x larger. Same query, same plan the optimizer wants to use — but now it can't fit the hash table in memory, so it spills, and your "identical" query behaves completely differently under identical SQL.

And this isn't unique to databases. It's the same shape of constraint you'll see anywhere a system has to work within bounded memory — worth being precise that it's the shape that repeats, not the exact mechanism. Google's Omni model caps video edits at around ten seconds — not because ten seconds is some meaningful creative limit, but because that's roughly what fits in its working context. The model's response to running out of context is truncation; a database's response to running out of RAM during a hash join is spilling to disk. Different fix, same root cause: a system quietly changing strategy the moment it outgrows the memory it was counting on — a cost you don't see until the moment it happens.

The code, reframed

Here's the exact query from the meme one more time — same SQL, but now the comments describe what's actually happening underneath instead of what the output looks like:

SELECT
    e.employee_id,      -- probe key: this column drives the hash table lookup
    e.employee_name,
    e.department,
    s.net_sales
FROM employees e
JOIN sales s USING (employee_id);
-- If `sales` is small: engine likely builds a hash table from `sales` in memory,
--   then streams `employees` past it in one pass — O(n + m).
-- If `employee_id` is indexed on both sides: engine may prefer a merge join,
--   walking both sorted B-trees in lockstep — no hash table needed at all.
-- If statistics are stale or tables are tiny: engine may fall back to nested loop —
--   fine here, disastrous if either table grows without you noticing.

You don't have to guess which one your database actually picked. You can just ask it:

EXPLAIN ANALYZE
SELECT e.employee_id, e.employee_name, e.department, s.net_sales
FROM employees e
JOIN sales s USING (employee_id);

Look for Hash Join, Merge Join, or Nested Loop in the output — that's your database telling you, in plain terms, which of the three strategies above it's actually running against your real data, right now. Here's what that actually looks like on a real (Postgres) run:

Hash Join  (cost=1.23..458.67 rows=1000 width=64) (actual time=0.045..3.201 rows=1000 loops=1)
  Hash Cond: (e.employee_id = s.employee_id)
  ->  Seq Scan on employees e  (cost=0.00..22.00 rows=1000 width=40)
  ->  Hash  (cost=1.15..1.15 rows=6 width=24)
        ->  Seq Scan on sales s  (cost=0.00..1.15 rows=6 width=24)
Planning Time: 0.112 ms
Execution Time: 3.245 ms

Read it from the bottom up: the engine scanned sales (Seq Scan on sales s), built a Hash from it, then joined that hash against a scan of employees — exactly the BUILD-then-PROBE pattern from the pseudocode above. If this were spilling to disk, you'd see it flagged explicitly further down the plan (Postgres reports Batches greater than 1, which means the hash table didn't fit in the memory it was given and got split up) — that's your direct, no-guessing confirmation that the memory-bound-to-I/O-bound flip we just walked through is actually happening on your data.

Where this leaves you

ON vs USING was never the real question. The real question is: what does your database know about your data — its size, its sort order, the memory it has to work with — and which of these three strategies does that knowledge point it toward?

Next time your query feels slow, skip the syntax tweaking. Run EXPLAIN ANALYZE and find out which of these three things is actually happening in memory. That's where the answer lives.