postgresql/src/backend/tsearch
Heikki Linnakangas 11854dee01 Reduce memory usage of tsvector type analyze function.
compute_tsvector_stats() detoasted and kept in memory every tsvector value
in the sample, but that can be a lot of memory. The original bug report
described a case using over 10 gigabytes, with statistics target of 10000
(the maximum).

To fix, allocate a separate copy of just the lexemes that we keep around,
and free the detoasted tsvector values as we go. This adds some palloc/pfree
overhead, when you have a lot of distinct lexemes in the sample, but it's
better than running out of memory.

Fixes bug #14654 reported by James C. Reviewed by Tom Lane. Backport to
all supported versions.

Discussion: https://www.postgresql.org/message-id/20170514200602.1451.46797@wrigleys.postgresql.org
2017-07-12 22:04:08 +03:00
..
dict.c Update copyrights for 2013 2013-01-01 17:15:01 -05:00
dict_ispell.c Avoid memcpy() with same source and destination address. 2014-03-07 13:29:19 +02:00
dict_simple.c Update copyrights for 2013 2013-01-01 17:15:01 -05:00
dict_synonym.c Update copyrights for 2013 2013-01-01 17:15:01 -05:00
dict_thesaurus.c Cope with more than 64K phrases in a thesaurus dictionary. 2014-11-06 20:52:52 -05:00
hunspell_sample.affix Add regression tests for ispell, synonym and thesaurus dictionaries. 2007-09-11 11:54:42 +00:00
ispell_sample.affix Add regression tests for ispell, synonym and thesaurus dictionaries. 2007-09-11 11:54:42 +00:00
ispell_sample.dict Add regression tests for ispell, synonym and thesaurus dictionaries. 2007-09-11 11:54:42 +00:00
Makefile Update copyrights for 2013 2013-01-01 17:15:01 -05:00
regis.c Update copyrights for 2013 2013-01-01 17:15:01 -05:00
spell.c Fix not-terribly-safe coding in NIImportOOAffixes() and NIImportAffixes(). 2016-03-06 19:21:03 -05:00
synonym_sample.syn Add prefix support for synonym dictionary 2009-08-14 14:53:20 +00:00
thesaurus_sample.ths Remove extra newlines at end and beginning of files, add missing newlines 2010-08-19 05:57:36 +00:00
to_tsany.c Use memmove() instead of memcpy() for copying overlapping regions. 2014-02-10 09:59:49 +02:00
ts_locale.c Remove tabs after spaces in C comments 2014-05-06 11:26:28 -04:00
ts_parse.c Fix typos in comments. 2017-02-06 11:34:28 +02:00
ts_selfuncs.c Remove tabs after spaces in C comments 2014-05-06 11:26:28 -04:00
ts_typanalyze.c Reduce memory usage of tsvector type analyze function. 2017-07-12 22:04:08 +03:00
ts_utils.c Remove tabs after spaces in C comments 2014-05-06 11:26:28 -04:00
wparser.c Update copyrights for 2013 2013-01-01 17:15:01 -05:00
wparser_def.c Fix typos in comments. 2017-02-06 11:34:28 +02:00