Run it
The three grouping idioms agreeing, groupby getting it wrong on unsorted input, and both inversion failures caught in the act.
The defaultdict trap
A defaultdict creates entries on read. Merely looking a key up inserts it:
That surprises people debugging a size or an iteration. Two ways to avoid it: use d.get(key, []) for reads, or convert to a plain dict once building is finished with dict(d).
The related habit worth having: never pass a defaultdict out of a function as a return value if callers will read arbitrary keys. Convert it first, and the mutation-on-read behaviour stays inside the code that wanted it.
Inverting a dictionary
Duplicate values collide, and the last one wins. alice and carol are both 30, so only carol survives. This is the single most common bug with inversion, and the fix is to invert into lists:
Which is the same operation as grouping — grouping the keys by their value. The two topics are one topic.
The other requirement: values must be hashable to become keys. Inverting {"a": [1, 2]} raises TypeError, and converting the list to a tuple first fixes it.
Three ways to group
setdefault(key, []).append(x) — no import, and it makes the default explicit. It does construct an empty list on every call, which is why the next form is usually preferred.
defaultdict(list) — the idiomatic answer. Remember that reading a missing key creates it, so it is not safe to inspect casually.
itertools.groupby — the one that catches people. It groups consecutive equal keys only, so it needs the input sorted by the same key first. It is not the SQL GROUP BY its name suggests.
Two ways inversion breaks
Duplicate values. {v: k for k, v in d.items()} keeps only the last key for each repeated value, silently. If duplicates are possible, invert to lists instead — which is just grouping again, by value.
Unhashable values. A value that was fine as a value cannot necessarily be a key: a dict mapping names to lists of scores cannot be inverted directly, because a list is unhashable.
Where this shows up
Grouping is the shape behind most "organise this data" questions: anagram groups keyed on sorted letters, files keyed on their hash, log lines keyed on their level. Recognising it saves you from writing a nested loop that scans for an existing group — the accidental O(n²) this idiom exists to avoid. The one-pass defaultdict build is O(n); the nested-loop version that re-scans the groups already built is O(n²), which on a hundred thousand records is the difference between ten thousand operations and ten billion — the same idiom, and the reason it is worth reaching for on sight.
The standard library has a groupby, and it does not do what the name suggests to anyone coming from SQL or pandas.
itertools.groupby groups consecutive runs only. It does not collect scattered items. To get SQL-like behaviour the input must be sorted by the same key first:
Which costs O(n log n), while the defaultdict version is O(n). Prefer defaultdict unless the data is already sorted or the runs themselves are what you want — compressing repeated values, detecting consecutive duplicates, run-length encoding.
The other sharp edge: the group is an iterator sharing the underlying stream. It is exhausted as soon as you advance to the next group, so list(group) must happen before moving on. groups = [(k, g) for k, g in groupby(x)] yields empty groups for exactly this reason.
The grouping patterns worth having ready
Group by a computed key.
Group by several keys — use a tuple:
Count instead of collect — Counter is the specialised form:
Aggregate while grouping — defaultdict(int) for sums, defaultdict(set) for deduplicated members:
Group anagrams, the classic interview form — the key is a canonical form:
groups = defaultdict(list)
for w in words:
groups["".join(sorted(w))].append(w)
return list(groups.values())
| Need | Tool |
|---|
| Collect items | defaultdict(list) |
| Count items | Counter |
| Sum values | defaultdict(int) |
| Deduplicated members | defaultdict(set) |
| Nested grouping | defaultdict(lambda: defaultdict(list)) |
| Consecutive runs only | itertools.groupby |
The nested case needs a lambda because defaultdict(defaultdict(list)) is wrong — the argument must be a callable that produces the default, not an instance.
Questions people ask
Why defaultdict over setdefault? Cleaner in a loop and slightly faster, since setdefault builds the default list on every call even when the key exists. Use setdefault when the result must be a plain dict.
What happens when inverting with duplicate values? Later keys overwrite earlier ones. Invert into lists to keep all of them.
Can any value become a key? Only hashable ones. Convert lists to tuples.
Why does itertools.groupby split my groups? It only groups consecutive items; sort by the key first.
How do I stop defaultdict inserting on read? Use .get(), or wrap in dict() when done building.
Is Counter a dict? Yes, a subclass — and it returns 0 for missing keys rather than raising, without inserting them.
Recap in one screen
defaultdict(list) is the grouping idiom; setdefault is the plain-dict equivalent.- A
defaultdict inserts keys on read — use .get() or convert with dict() before handing it out. - Inverting with a comprehension silently drops duplicate values; invert into lists instead.
- Values becoming keys must be hashable.
itertools.groupby groups only consecutive runs and its groups are single-use iterators; sort first, or use defaultdict.