1996-10-27 04:55:05 -05:00
|
|
|
#-------------------------------------------------------------------------
|
|
|
|
|
#
|
|
|
|
|
# Makefile--
|
|
|
|
|
# Makefile for executor
|
|
|
|
|
#
|
|
|
|
|
# IDENTIFICATION
|
2010-09-20 16:08:53 -04:00
|
|
|
# src/backend/executor/Makefile
|
1996-10-27 04:55:05 -05:00
|
|
|
#
|
|
|
|
|
#-------------------------------------------------------------------------
|
|
|
|
|
|
2000-08-31 12:12:35 -04:00
|
|
|
subdir = src/backend/executor
|
|
|
|
|
top_builddir = ../../..
|
|
|
|
|
include $(top_builddir)/src/Makefile.global
|
1996-10-27 04:55:05 -05:00
|
|
|
|
2019-11-05 17:41:07 -05:00
|
|
|
OBJS = \
|
|
|
|
|
execAmi.o \
|
Add support for asynchronous execution.
This implements asynchronous execution, which runs multiple parts of a
non-parallel-aware Append concurrently rather than serially to improve
performance when possible. Currently, the only node type that can be
run concurrently is a ForeignScan that is an immediate child of such an
Append. In the case where such ForeignScans access data on different
remote servers, this would run those ForeignScans concurrently, and
overlap the remote operations to be performed simultaneously, so it'll
improve the performance especially when the operations involve
time-consuming ones such as remote join and remote aggregation.
We may extend this to other node types such as joins or aggregates over
ForeignScans in the future.
This also adds the support for postgres_fdw, which is enabled by the
table-level/server-level option "async_capable". The default is false.
Robert Haas, Kyotaro Horiguchi, Thomas Munro, and myself. This commit
is mostly based on the patch proposed by Robert Haas, but also uses
stuff from the patch proposed by Kyotaro Horiguchi and from the patch
proposed by Thomas Munro. Reviewed by Kyotaro Horiguchi, Konstantin
Knizhnik, Andrey Lepikhov, Movead Li, Thomas Munro, Justin Pryzby, and
others.
Discussion: https://postgr.es/m/CA%2BTgmoaXQEt4tZ03FtQhnzeDEMzBck%2BLrni0UWHVVgOTnA6C1w%40mail.gmail.com
Discussion: https://postgr.es/m/CA%2BhUKGLBRyu0rHrDCMC4%3DRn3252gogyp1SjOgG8SEKKZv%3DFwfQ%40mail.gmail.com
Discussion: https://postgr.es/m/20200228.170650.667613673625155850.horikyota.ntt%40gmail.com
2021-03-31 05:45:00 -04:00
|
|
|
execAsync.o \
|
2019-11-05 17:41:07 -05:00
|
|
|
execCurrent.o \
|
|
|
|
|
execExpr.o \
|
|
|
|
|
execExprInterp.o \
|
|
|
|
|
execGrouping.o \
|
|
|
|
|
execIndexing.o \
|
|
|
|
|
execJunk.o \
|
|
|
|
|
execMain.o \
|
|
|
|
|
execParallel.o \
|
|
|
|
|
execPartition.o \
|
|
|
|
|
execProcnode.o \
|
|
|
|
|
execReplication.o \
|
|
|
|
|
execSRF.o \
|
|
|
|
|
execScan.o \
|
|
|
|
|
execTuples.o \
|
|
|
|
|
execUtils.o \
|
|
|
|
|
functions.o \
|
|
|
|
|
instrument.o \
|
|
|
|
|
nodeAgg.o \
|
|
|
|
|
nodeAppend.o \
|
|
|
|
|
nodeBitmapAnd.o \
|
|
|
|
|
nodeBitmapHeapscan.o \
|
|
|
|
|
nodeBitmapIndexscan.o \
|
|
|
|
|
nodeBitmapOr.o \
|
|
|
|
|
nodeCtescan.o \
|
|
|
|
|
nodeCustom.o \
|
|
|
|
|
nodeForeignscan.o \
|
|
|
|
|
nodeFunctionscan.o \
|
|
|
|
|
nodeGather.o \
|
|
|
|
|
nodeGatherMerge.o \
|
|
|
|
|
nodeGroup.o \
|
|
|
|
|
nodeHash.o \
|
|
|
|
|
nodeHashjoin.o \
|
Implement Incremental Sort
Incremental Sort is an optimized variant of multikey sort for cases when
the input is already sorted by a prefix of the requested sort keys. For
example when the relation is already sorted by (key1, key2) and we need
to sort it by (key1, key2, key3) we can simply split the input rows into
groups having equal values in (key1, key2), and only sort/compare the
remaining column key3.
This has a number of benefits:
- Reduced memory consumption, because only a single group (determined by
values in the sorted prefix) needs to be kept in memory. This may also
eliminate the need to spill to disk.
- Lower startup cost, because Incremental Sort produce results after each
prefix group, which is beneficial for plans where startup cost matters
(like for example queries with LIMIT clause).
We consider both Sort and Incremental Sort, and decide based on costing.
The implemented algorithm operates in two different modes:
- Fetching a minimum number of tuples without check of equality on the
prefix keys, and sorting on all columns when safe.
- Fetching all tuples for a single prefix group and then sorting by
comparing only the remaining (non-prefix) keys.
We always start in the first mode, and employ a heuristic to switch into
the second mode if we believe it's beneficial - the goal is to minimize
the number of unnecessary comparions while keeping memory consumption
below work_mem.
This is a very old patch series. The idea was originally proposed by
Alexander Korotkov back in 2013, and then revived in 2017. In 2018 the
patch was taken over by James Coleman, who wrote and rewrote most of the
current code.
There were many reviewers/contributors since 2013 - I've done my best to
pick the most active ones, and listed them in this commit message.
Author: James Coleman, Alexander Korotkov
Reviewed-by: Tomas Vondra, Andreas Karlsson, Marti Raudsepp, Peter Geoghegan, Robert Haas, Thomas Munro, Antonin Houska, Andres Freund, Alexander Kuzmenkov
Discussion: https://postgr.es/m/CAPpHfdscOX5an71nHd8WSUH6GNOCf=V7wgDaTXdDd9=goN-gfA@mail.gmail.com
Discussion: https://postgr.es/m/CAPpHfds1waRZ=NOmueYq0sx1ZSCnt+5QJvizT8ndT2=etZEeAQ@mail.gmail.com
2020-04-06 15:33:28 -04:00
|
|
|
nodeIncrementalSort.o \
|
2019-11-05 17:41:07 -05:00
|
|
|
nodeIndexonlyscan.o \
|
|
|
|
|
nodeIndexscan.o \
|
|
|
|
|
nodeLimit.o \
|
|
|
|
|
nodeLockRows.o \
|
|
|
|
|
nodeMaterial.o \
|
|
|
|
|
nodeMergeAppend.o \
|
|
|
|
|
nodeMergejoin.o \
|
|
|
|
|
nodeModifyTable.o \
|
|
|
|
|
nodeNamedtuplestorescan.o \
|
|
|
|
|
nodeNestloop.o \
|
|
|
|
|
nodeProjectSet.o \
|
|
|
|
|
nodeRecursiveunion.o \
|
|
|
|
|
nodeResult.o \
|
Add Result Cache executor node (take 2)
Here we add a new executor node type named "Result Cache". The planner
can include this node type in the plan to have the executor cache the
results from the inner side of parameterized nested loop joins. This
allows caching of tuples for sets of parameters so that in the event that
the node sees the same parameter values again, it can just return the
cached tuples instead of rescanning the inner side of the join all over
again. Internally, result cache uses a hash table in order to quickly
find tuples that have been previously cached.
For certain data sets, this can significantly improve the performance of
joins. The best cases for using this new node type are for join problems
where a large portion of the tuples from the inner side of the join have
no join partner on the outer side of the join. In such cases, hash join
would have to hash values that are never looked up, thus bloating the hash
table and possibly causing it to multi-batch. Merge joins would have to
skip over all of the unmatched rows. If we use a nested loop join with a
result cache, then we only cache tuples that have at least one join
partner on the outer side of the join. The benefits of using a
parameterized nested loop with a result cache increase when there are
fewer distinct values being looked up and the number of lookups of each
value is large. Also, hash probes to lookup the cache can be much faster
than the hash probe in a hash join as it's common that the result cache's
hash table is much smaller than the hash join's due to result cache only
caching useful tuples rather than all tuples from the inner side of the
join. This variation in hash probe performance is more significant when
the hash join's hash table no longer fits into the CPU's L3 cache, but the
result cache's hash table does. The apparent "random" access of hash
buckets with each hash probe can cause a poor L3 cache hit ratio for large
hash tables. Smaller hash tables generally perform better.
The hash table used for the cache limits itself to not exceeding work_mem
* hash_mem_multiplier in size. We maintain a dlist of keys for this cache
and when we're adding new tuples and realize we've exceeded the memory
budget, we evict cache entries starting with the least recently used ones
until we have enough memory to add the new tuples to the cache.
For parameterized nested loop joins, we now consider using one of these
result cache nodes in between the nested loop node and its inner node. We
determine when this might be useful based on cost, which is primarily
driven off of what the expected cache hit ratio will be. Estimating the
cache hit ratio relies on having good distinct estimates on the nested
loop's parameters.
For now, the planner will only consider using a result cache for
parameterized nested loop joins. This works for both normal joins and
also for LATERAL type joins to subqueries. It is possible to use this new
node for other uses in the future. For example, to cache results from
correlated subqueries. However, that's not done here due to some
difficulties obtaining a distinct estimation on the outer plan to
calculate the estimated cache hit ratio. Currently we plan the inner plan
before planning the outer plan so there is no good way to know if a result
cache would be useful or not since we can't estimate the number of times
the subplan will be called until the outer plan is generated.
The functionality being added here is newly introducing a dependency on
the return value of estimate_num_groups() during the join search.
Previously, during the join search, we only ever needed to perform
selectivity estimations. With this commit, we need to use
estimate_num_groups() in order to estimate what the hit ratio on the
result cache will be. In simple terms, if we expect 10 distinct values
and we expect 1000 outer rows, then we'll estimate the hit ratio to be
99%. Since cache hits are very cheap compared to scanning the underlying
nodes on the inner side of the nested loop join, then this will
significantly reduce the planner's cost for the join. However, it's
fairly easy to see here that things will go bad when estimate_num_groups()
incorrectly returns a value that's significantly lower than the actual
number of distinct values. If this happens then that may cause us to make
use of a nested loop join with a result cache instead of some other join
type, such as a merge or hash join. Our distinct estimations have been
known to be a source of trouble in the past, so the extra reliance on them
here could cause the planner to choose slower plans than it did previous
to having this feature. Distinct estimations are also fairly hard to
estimate accurately when several tables have been joined already or when a
WHERE clause filters out a set of values that are correlated to the
expressions we're estimating the number of distinct value for.
For now, the costing we perform during query planning for result caches
does put quite a bit of faith in the distinct estimations being accurate.
When these are accurate then we should generally see faster execution
times for plans containing a result cache. However, in the real world, we
may find that we need to either change the costings to put less trust in
the distinct estimations being accurate or perhaps even disable this
feature by default. There's always an element of risk when we teach the
query planner to do new tricks that it decides to use that new trick at
the wrong time and causes a regression. Users may opt to get the old
behavior by turning the feature off using the enable_resultcache GUC.
Currently, this is enabled by default. It remains to be seen if we'll
maintain that setting for the release.
Additionally, the name "Result Cache" is the best name I could think of
for this new node at the time I started writing the patch. Nobody seems
to strongly dislike the name. A few people did suggest other names but no
other name seemed to dominate in the brief discussion that there was about
names. Let's allow the beta period to see if the current name pleases
enough people. If there's some consensus on a better name, then we can
change it before the release. Please see the 2nd discussion link below
for the discussion on the "Result Cache" name.
Author: David Rowley
Reviewed-by: Andy Fan, Justin Pryzby, Zhihong Yu, Hou Zhijie
Tested-By: Konstantin Knizhnik
Discussion: https://postgr.es/m/CAApHDvrPcQyQdWERGYWx8J%2B2DLUNgXu%2BfOSbQ1UscxrunyXyrQ%40mail.gmail.com
Discussion: https://postgr.es/m/CAApHDvq=yQXr5kqhRviT2RhNKwToaWr9JAN5t+5_PzhuRJ3wvg@mail.gmail.com
2021-04-01 21:10:56 -04:00
|
|
|
nodeResultCache.o \
|
2019-11-05 17:41:07 -05:00
|
|
|
nodeSamplescan.o \
|
|
|
|
|
nodeSeqscan.o \
|
|
|
|
|
nodeSetOp.o \
|
|
|
|
|
nodeSort.o \
|
|
|
|
|
nodeSubplan.o \
|
|
|
|
|
nodeSubqueryscan.o \
|
|
|
|
|
nodeTableFuncscan.o \
|
2021-02-27 04:59:36 -05:00
|
|
|
nodeTidrangescan.o \
|
2019-11-05 17:41:07 -05:00
|
|
|
nodeTidscan.o \
|
|
|
|
|
nodeUnique.o \
|
|
|
|
|
nodeValuesscan.o \
|
|
|
|
|
nodeWindowAgg.o \
|
|
|
|
|
nodeWorktablescan.o \
|
|
|
|
|
spi.o \
|
|
|
|
|
tqueue.o \
|
|
|
|
|
tstoreReceiver.o
|
1996-10-27 04:55:05 -05:00
|
|
|
|
2008-02-19 05:30:09 -05:00
|
|
|
include $(top_srcdir)/src/backend/common.mk
|
