Rez caching

One-sentence summary: rez has two caches that share almost nothing — a memcached resolve cache that stores solves and package file reads, and a local-disk package cache that stores variant payloads — and the single most useful move when a change seems to be ignored is to disable one at a time and see which one was lying.

Skill scope: what is cached, when it goes stale, and how to prove it. For the solver internals that a cached resolve short-circuits see rez-resolve; for the six causes of a failed resolve see rez-resolve-troubleshooting; for a package that fails to load or was left half-installed see rez-package-pitfalls; for the cachable attribute in a package.py see rez-package-definition.

The two caches

They are configured separately, live in different places, and fail in different ways. Naming the right one first is most of the work.

Both are off out of the box, so a cache that is biting you is one somebody turned on. Check before you theorise:

rez-config memcached_uri
rez-config cache_packages_path

An empty list and None respectively mean neither cache is active.

Resolve caching

Backed by memcached. In a studio, a machine that performs a solve another machine already performed gets the cached result instead of re-solving.

# rezconfig.py
memcached_uri = ["127.0.0.1:11211"]

That one setting is all that is required. The related switches:

resolve_caching = True with memcached_uri = [] is not caching. The setting is a permission slip, not a switch — the URI list is what turns it on.

What invalidation actually covers

Each cached solve stores solver state, the timestamps of the packages it saw, and the variant state of every variant involved. For the filesystem repository the variant state is the last modified time of the file backing the variant. Two consequences:

• Releasing a newer version that would change the result does invalidate the entry. A cached foo-1+<2 pointing at 1.0.0 is discarded once 1.0.1 exists.
• Changing a variant's backing file does invalidate the entry, because its state changed.

The gap is the case upstream calls out: "hacking" a released package into production in a way that does not produce a new version and does not move the file rez is watching. That entry is not invalidated, and you get the old answer.

rez-memcache

rez-memcache --stats             # hit/miss ratios and memory per server
rez-memcache --flush             # drop every cache entry
rez-memcache --reset-stats       # zero the counters, keep the entries
rez-memcache --warm              # pre-populate with visible packages
rez-memcache --poll              # continuously show get/sets per second

--stats is the one to reach for first — a high hit ratio on a studio server is the whole point of the cache, and a ratio near zero means it is not doing anything.

With no servers configured — the default, since memcached_uri is empty — every rez-memcache subcommand prints memcaching is not enabled. to stderr and exits 1. That is worth knowing before you put it in a script: exit 1 here means "caching is off", which is the normal state, not a fault. If you only care whether memcaching is configured, redirect stderr and treat 1 as informational:

rez-memcache --stats 2>&1 || true

--flush is the blunt instrument. It is safe — the cache is a cache — but on a busy studio server it pushes the cost of every subsequent solve back onto the machines that were sharing results.

To see memcached traffic per operation, set REZ_DEBUG_MEMCACHE (or debug_memcache) for everything, or debug_resolve_memcache for the resolve cache alone.

Package caching

Copies variant payloads from shared storage onto local disk so that running, say, a Python process does not pull every source file across the network.

Package caching does not cache package definitions — only payloads.

# rezconfig.py
cache_packages_path = "/path/to/local/cache"

Individual packages opt in with the cachable attribute; when it is absent, rez falls back to default_cachable_per_repository, then default_cachable_per_package, then default_cachable (False).

Cached variants are assumed immutable

This is the setting people get wrong. No check is ever made to see whether a variant's payload changed and should replace an existing cache entry. Caching a repository whose packages get overwritten in place will serve the old payload forever.

It is for exactly this reason that caching is off for local packages by default (package_cache_local), and for a source on the same disk as the cache (package_cache_same_device). Both are for testing only.

Asynchronous, so the first resolve does not benefit

With package_cache_async = True (the default), variants are copied by a separate rez-pkg-cache process after the resolve returns. A resolve will therefore often not use the cached copies it should, the first time around. Package caching is cumulative by design: more cached variants get used over time. That is a deliberate trade — it avoids blocking every resolve on a network copy.

To make a resolve wait, use rez-pkg-cache --pkg-cache-mode sync.

rez-pkg-cache

rez-pkg-cache                            # table of every cached variant
rez-pkg-cache -c status package variant_uri
rez-pkg-cache --logs                     # what the cache daemon did
rez-pkg-cache --clean                    # delete stalled and unused variants

Each variant is one of three statuses, and the status tells you what to do next:

You can see which variants a resolve actually used — the (cached) label in the right-hand column:

rez-env foo

Cached packages also record where the payload really lives, in REZ_<PKG>_ORIG_ROOT, while REZ_<PKG>_ROOT points at the cache. A package whose root is the cache is the fastest way to confirm caching is in effect.

Removing a variant takes two steps, and people routinely do only the first:

rez-pkg-cache -r <variant_uri>     # uncache it — the files stay on disk
rez-pkg-cache --clean              # purge the now-unused files

A stalled variant will never be re-cached until a --clean removes it, and package_cache_clean_limit deliberately will not clean stalled entries either — otherwise a problematic variant could cycle cached → stalled → deleted → cached forever. --clean is the only thing that removes a stalled variant.

Without package_cache_clean_limit (or a scheduled rez-pkg-cache --clean), the cache grows indefinitely.

"I changed the package and nothing took effect"

Walk this in order. Each step names the cache it rules out, so you stop at the one that was the cause instead of flushing everything.

1. Is the change even visible? rez-search foo. If the version you edited is not there, this is not a caching problem — it is a build/release problem, or a failed build that left a broken install. See rez-package-pitfalls.
2. Bypass the resolve cache. rez-env foo --no-cache. If the change appears, the resolve cache served a stale entry. Flush it with rez-memcache --flush and re-check.
3. Bypass the package cache. rez-env foo --no-pkg-cache. If the change appears here but not in step 2, the payload cache is serving an old copy — most likely because the package was overwritten in place rather than released as a new version. Uncache and clean it: rez-pkg-cache -r <variant_uri> then rez-pkg-cache --clean.
4. Confirm in a fresh process, not the build shell. rez-env foo -o context.rxt then rez-context --so. A green build proves the file parses, not that a later resolve picks it up.

If the change appears with both caches bypassed but not without them, and neither flush helps, the entry is being re-populated by something else — usually another machine, or a rez-config override you did not expect. Check where the settings come from:

rez-config --source-list resolve_caching

Measuring instead of guessing

rez-benchmark runs a fixed set of resolves and writes a results directory containing summary.json. Use it before and after a cache change rather than trusting a stopwatch.

Run each benchmark on its own — --compare replaces the benchmark run rather than following it, so a second rez-benchmark --out after --compare before never writes after at all. Two separate runs, then the comparison as a third command:

rez-benchmark --out before
rez-benchmark --out after
rez-benchmark --out after --compare before
rez-benchmark --histogram --out after

--compare reports each statistic as a [delta, percentage] pair, and the sign is easy to read backwards. The delta is always the --compare directory minus the --out directory:

mean_delta = mean(--compare) - mean(--out)

So for the command above, where --out is after and --compare is before:

• mean_delta is positive → after is faster (the baseline was slower, so your change helped).
• mean_delta is negative → after is slower (your change made things worse).

Say which directory is which rather than relying on "the second one" — the two directories play opposite roles, and the whole point of the run is one number's sign.

One caveat: the comparison divides by the --out summary's values, so a --out summary containing a 0.0 (a min of zero is common on a fast machine) fails with ZeroDivisionError. Swapping the two directories usually clears it, since the other run rarely has a zero in the same field.

For a per-solve breakdown, rez-env --stats prints solver counters (num_solves, load_time, solve_time, reduction and intersection counts). It reports solver statistics, not cache hit ratios — use rez-memcache --stats for those.

Troubleshooting

Agent workflow

1. Establish which caches are even on: rez-config memcached_uri and rez-config cache_packages_path.
2. For "my change did not take effect", first confirm the change is visible with rez-search.
3. Bypass one cache at a time — --no-cache, then --no-pkg-cache — and stop at the one that changes the answer.
4. Fix the specific cache (rez-memcache --flush, or rez-pkg-cache -r + --clean) rather than flushing both.
5. Verify in a fresh resolve (rez-env -o + rez-context --so), not in the shell that built it.
6. For slowness claims, measure with rez-benchmark --compare before and after the change.