fixempties() counts the number of in-arcs in the regex NFA and then
allocates an array of that many arc pointers. If the NFA contains no
arcs, this amounts to malloc(0) for which some platforms return NULL.
The code mistakenly treats that as indicating out-of-memory. Thus,
we can get a bogus "out of memory" failure for some unsatisfiable
regexes.
This happens only in v15 and earlier, since bea3d7e38 switched to
using palloc() rather than bare malloc(). And at least of the
platforms in the buildfarm, only AIX seems to return NULL. So the
impact is pretty narrow. But I don't especially want to ship code
that is failing its own regression tests, so let's fix this for
this week's releases.
A quick code survey says that there is only the one place in
src/backend/regex/ that is at risk of doing malloc(0), so we'll just
band-aid that place. A more future-proof fix could be to install a
malloc() wrapper similar to pg_malloc(). But this code seems unlikely
to change much more in the affected branches, so that's probably
overkill.
The only known test case for this involves a complemented character
class in a bracket expression, for example [^\s\S], and we didn't
support that in v13. So it may be that the problem is unreachable
in v13. But I'm not 100% sure of that, so patch v13 too.
Discussion: https://postgr.es/m/661fd81b-f069-8f30-1a41-e195c57449b4@gmail.com
df1a2633b1