1996-07-09 02:22:35 -04:00
|
|
|
/*-------------------------------------------------------------------------
|
|
|
|
|
*
|
1999-02-13 18:22:53 -05:00
|
|
|
* nodeHash.h
|
2003-01-10 18:54:24 -05:00
|
|
|
* prototypes for nodeHash.c
|
1996-07-09 02:22:35 -04:00
|
|
|
*
|
|
|
|
|
*
|
2024-01-03 20:49:05 -05:00
|
|
|
* Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
|
2000-01-26 00:58:53 -05:00
|
|
|
* Portions Copyright (c) 1994, Regents of the University of California
|
1996-07-09 02:22:35 -04:00
|
|
|
*
|
2010-09-20 16:08:53 -04:00
|
|
|
* src/include/executor/nodeHash.h
|
1996-07-09 02:22:35 -04:00
|
|
|
*
|
|
|
|
|
*-------------------------------------------------------------------------
|
|
|
|
|
*/
|
1997-09-07 01:04:48 -04:00
|
|
|
#ifndef NODEHASH_H
|
|
|
|
|
#define NODEHASH_H
|
1996-07-09 02:22:35 -04:00
|
|
|
|
2017-12-05 13:55:56 -05:00
|
|
|
#include "access/parallel.h"
|
2002-12-05 10:50:39 -05:00
|
|
|
#include "nodes/execnodes.h"
|
1998-02-25 23:46:47 -05:00
|
|
|
|
Add parallel-aware hash joins.
Introduce parallel-aware hash joins that appear in EXPLAIN plans as Parallel
Hash Join with Parallel Hash. While hash joins could already appear in
parallel queries, they were previously always parallel-oblivious and had a
partial subplan only on the outer side, meaning that the work of the inner
subplan was duplicated in every worker.
After this commit, the planner will consider using a partial subplan on the
inner side too, using the Parallel Hash node to divide the work over the
available CPU cores and combine its results in shared memory. If the join
needs to be split into multiple batches in order to respect work_mem, then
workers process different batches as much as possible and then work together
on the remaining batches.
The advantages of a parallel-aware hash join over a parallel-oblivious hash
join used in a parallel query are that it:
* avoids wasting memory on duplicated hash tables
* avoids wasting disk space on duplicated batch files
* divides the work of building the hash table over the CPUs
One disadvantage is that there is some communication between the participating
CPUs which might outweigh the benefits of parallelism in the case of small
hash tables. This is avoided by the planner's existing reluctance to supply
partial plans for small scans, but it may be necessary to estimate
synchronization costs in future if that situation changes. Another is that
outer batch 0 must be written to disk if multiple batches are required.
A potential future advantage of parallel-aware hash joins is that right and
full outer joins could be supported, since there is a single set of matched
bits for each hashtable, but that is not yet implemented.
A new GUC enable_parallel_hash is defined to control the feature, defaulting
to on.
Author: Thomas Munro
Reviewed-By: Andres Freund, Robert Haas
Tested-By: Rafia Sabih, Prabhat Sahu
Discussion:
https://postgr.es/m/CAEepm=2W=cOkiZxcg6qiFQP-dHUe09aqTrEMM7yJDrHMhDv_RA@mail.gmail.com
https://postgr.es/m/CAEepm=37HKyJ4U6XOLi=JgfSHM3o6B-GaeO-6hkOmneTDkH+Uw@mail.gmail.com
2017-12-21 02:39:21 -05:00
|
|
|
struct SharedHashJoinBatch;
|
|
|
|
|
|
2006-02-27 23:10:28 -05:00
|
|
|
extern HashState *ExecInitHash(Hash *node, EState *estate, int eflags);
|
2005-04-16 16:07:35 -04:00
|
|
|
extern Node *MultiExecHash(HashState *node);
|
2002-12-05 10:50:39 -05:00
|
|
|
extern void ExecEndHash(HashState *node);
|
2010-07-12 13:01:06 -04:00
|
|
|
extern void ExecReScanHash(HashState *node);
|
2002-12-05 10:50:39 -05:00
|
|
|
|
2019-03-22 07:09:32 -04:00
|
|
|
extern HashJoinTable ExecHashTableCreate(HashState *state, List *hashOperators, List *hashCollations,
|
2019-05-22 13:04:48 -04:00
|
|
|
bool keepNulls);
|
Add parallel-aware hash joins.
Introduce parallel-aware hash joins that appear in EXPLAIN plans as Parallel
Hash Join with Parallel Hash. While hash joins could already appear in
parallel queries, they were previously always parallel-oblivious and had a
partial subplan only on the outer side, meaning that the work of the inner
subplan was duplicated in every worker.
After this commit, the planner will consider using a partial subplan on the
inner side too, using the Parallel Hash node to divide the work over the
available CPU cores and combine its results in shared memory. If the join
needs to be split into multiple batches in order to respect work_mem, then
workers process different batches as much as possible and then work together
on the remaining batches.
The advantages of a parallel-aware hash join over a parallel-oblivious hash
join used in a parallel query are that it:
* avoids wasting memory on duplicated hash tables
* avoids wasting disk space on duplicated batch files
* divides the work of building the hash table over the CPUs
One disadvantage is that there is some communication between the participating
CPUs which might outweigh the benefits of parallelism in the case of small
hash tables. This is avoided by the planner's existing reluctance to supply
partial plans for small scans, but it may be necessary to estimate
synchronization costs in future if that situation changes. Another is that
outer batch 0 must be written to disk if multiple batches are required.
A potential future advantage of parallel-aware hash joins is that right and
full outer joins could be supported, since there is a single set of matched
bits for each hashtable, but that is not yet implemented.
A new GUC enable_parallel_hash is defined to control the feature, defaulting
to on.
Author: Thomas Munro
Reviewed-By: Andres Freund, Robert Haas
Tested-By: Rafia Sabih, Prabhat Sahu
Discussion:
https://postgr.es/m/CAEepm=2W=cOkiZxcg6qiFQP-dHUe09aqTrEMM7yJDrHMhDv_RA@mail.gmail.com
https://postgr.es/m/CAEepm=37HKyJ4U6XOLi=JgfSHM3o6B-GaeO-6hkOmneTDkH+Uw@mail.gmail.com
2017-12-21 02:39:21 -05:00
|
|
|
extern void ExecParallelHashTableAlloc(HashJoinTable hashtable,
|
2019-05-22 13:04:48 -04:00
|
|
|
int batchno);
|
1997-09-07 22:41:22 -04:00
|
|
|
extern void ExecHashTableDestroy(HashJoinTable hashtable);
|
Add parallel-aware hash joins.
Introduce parallel-aware hash joins that appear in EXPLAIN plans as Parallel
Hash Join with Parallel Hash. While hash joins could already appear in
parallel queries, they were previously always parallel-oblivious and had a
partial subplan only on the outer side, meaning that the work of the inner
subplan was duplicated in every worker.
After this commit, the planner will consider using a partial subplan on the
inner side too, using the Parallel Hash node to divide the work over the
available CPU cores and combine its results in shared memory. If the join
needs to be split into multiple batches in order to respect work_mem, then
workers process different batches as much as possible and then work together
on the remaining batches.
The advantages of a parallel-aware hash join over a parallel-oblivious hash
join used in a parallel query are that it:
* avoids wasting memory on duplicated hash tables
* avoids wasting disk space on duplicated batch files
* divides the work of building the hash table over the CPUs
One disadvantage is that there is some communication between the participating
CPUs which might outweigh the benefits of parallelism in the case of small
hash tables. This is avoided by the planner's existing reluctance to supply
partial plans for small scans, but it may be necessary to estimate
synchronization costs in future if that situation changes. Another is that
outer batch 0 must be written to disk if multiple batches are required.
A potential future advantage of parallel-aware hash joins is that right and
full outer joins could be supported, since there is a single set of matched
bits for each hashtable, but that is not yet implemented.
A new GUC enable_parallel_hash is defined to control the feature, defaulting
to on.
Author: Thomas Munro
Reviewed-By: Andres Freund, Robert Haas
Tested-By: Rafia Sabih, Prabhat Sahu
Discussion:
https://postgr.es/m/CAEepm=2W=cOkiZxcg6qiFQP-dHUe09aqTrEMM7yJDrHMhDv_RA@mail.gmail.com
https://postgr.es/m/CAEepm=37HKyJ4U6XOLi=JgfSHM3o6B-GaeO-6hkOmneTDkH+Uw@mail.gmail.com
2017-12-21 02:39:21 -05:00
|
|
|
extern void ExecHashTableDetach(HashJoinTable hashtable);
|
|
|
|
|
extern void ExecHashTableDetachBatch(HashJoinTable hashtable);
|
|
|
|
|
extern void ExecParallelHashTableSetCurrentBatch(HashJoinTable hashtable,
|
2019-05-22 13:04:48 -04:00
|
|
|
int batchno);
|
Add parallel-aware hash joins.
Introduce parallel-aware hash joins that appear in EXPLAIN plans as Parallel
Hash Join with Parallel Hash. While hash joins could already appear in
parallel queries, they were previously always parallel-oblivious and had a
partial subplan only on the outer side, meaning that the work of the inner
subplan was duplicated in every worker.
After this commit, the planner will consider using a partial subplan on the
inner side too, using the Parallel Hash node to divide the work over the
available CPU cores and combine its results in shared memory. If the join
needs to be split into multiple batches in order to respect work_mem, then
workers process different batches as much as possible and then work together
on the remaining batches.
The advantages of a parallel-aware hash join over a parallel-oblivious hash
join used in a parallel query are that it:
* avoids wasting memory on duplicated hash tables
* avoids wasting disk space on duplicated batch files
* divides the work of building the hash table over the CPUs
One disadvantage is that there is some communication between the participating
CPUs which might outweigh the benefits of parallelism in the case of small
hash tables. This is avoided by the planner's existing reluctance to supply
partial plans for small scans, but it may be necessary to estimate
synchronization costs in future if that situation changes. Another is that
outer batch 0 must be written to disk if multiple batches are required.
A potential future advantage of parallel-aware hash joins is that right and
full outer joins could be supported, since there is a single set of matched
bits for each hashtable, but that is not yet implemented.
A new GUC enable_parallel_hash is defined to control the feature, defaulting
to on.
Author: Thomas Munro
Reviewed-By: Andres Freund, Robert Haas
Tested-By: Rafia Sabih, Prabhat Sahu
Discussion:
https://postgr.es/m/CAEepm=2W=cOkiZxcg6qiFQP-dHUe09aqTrEMM7yJDrHMhDv_RA@mail.gmail.com
https://postgr.es/m/CAEepm=37HKyJ4U6XOLi=JgfSHM3o6B-GaeO-6hkOmneTDkH+Uw@mail.gmail.com
2017-12-21 02:39:21 -05:00
|
|
|
|
2000-07-11 22:37:39 -04:00
|
|
|
extern void ExecHashTableInsert(HashJoinTable hashtable,
|
2019-05-22 13:04:48 -04:00
|
|
|
TupleTableSlot *slot,
|
|
|
|
|
uint32 hashvalue);
|
Add parallel-aware hash joins.
Introduce parallel-aware hash joins that appear in EXPLAIN plans as Parallel
Hash Join with Parallel Hash. While hash joins could already appear in
parallel queries, they were previously always parallel-oblivious and had a
partial subplan only on the outer side, meaning that the work of the inner
subplan was duplicated in every worker.
After this commit, the planner will consider using a partial subplan on the
inner side too, using the Parallel Hash node to divide the work over the
available CPU cores and combine its results in shared memory. If the join
needs to be split into multiple batches in order to respect work_mem, then
workers process different batches as much as possible and then work together
on the remaining batches.
The advantages of a parallel-aware hash join over a parallel-oblivious hash
join used in a parallel query are that it:
* avoids wasting memory on duplicated hash tables
* avoids wasting disk space on duplicated batch files
* divides the work of building the hash table over the CPUs
One disadvantage is that there is some communication between the participating
CPUs which might outweigh the benefits of parallelism in the case of small
hash tables. This is avoided by the planner's existing reluctance to supply
partial plans for small scans, but it may be necessary to estimate
synchronization costs in future if that situation changes. Another is that
outer batch 0 must be written to disk if multiple batches are required.
A potential future advantage of parallel-aware hash joins is that right and
full outer joins could be supported, since there is a single set of matched
bits for each hashtable, but that is not yet implemented.
A new GUC enable_parallel_hash is defined to control the feature, defaulting
to on.
Author: Thomas Munro
Reviewed-By: Andres Freund, Robert Haas
Tested-By: Rafia Sabih, Prabhat Sahu
Discussion:
https://postgr.es/m/CAEepm=2W=cOkiZxcg6qiFQP-dHUe09aqTrEMM7yJDrHMhDv_RA@mail.gmail.com
https://postgr.es/m/CAEepm=37HKyJ4U6XOLi=JgfSHM3o6B-GaeO-6hkOmneTDkH+Uw@mail.gmail.com
2017-12-21 02:39:21 -05:00
|
|
|
extern void ExecParallelHashTableInsert(HashJoinTable hashtable,
|
|
|
|
|
TupleTableSlot *slot,
|
|
|
|
|
uint32 hashvalue);
|
2019-05-22 13:04:48 -04:00
|
|
|
extern void ExecParallelHashTableInsertCurrentBatch(HashJoinTable hashtable,
|
|
|
|
|
TupleTableSlot *slot,
|
|
|
|
|
uint32 hashvalue);
|
2007-01-28 18:21:26 -05:00
|
|
|
extern bool ExecHashGetHashValue(HashJoinTable hashtable,
|
2019-05-22 13:04:48 -04:00
|
|
|
ExprContext *econtext,
|
|
|
|
|
List *hashkeys,
|
|
|
|
|
bool outer_tuple,
|
|
|
|
|
bool keep_nulls,
|
|
|
|
|
uint32 *hashvalue);
|
2005-03-06 17:15:05 -05:00
|
|
|
extern void ExecHashGetBucketAndBatch(HashJoinTable hashtable,
|
2019-05-22 13:04:48 -04:00
|
|
|
uint32 hashvalue,
|
|
|
|
|
int *bucketno,
|
|
|
|
|
int *batchno);
|
2010-12-30 20:24:55 -05:00
|
|
|
extern bool ExecScanHashBucket(HashJoinState *hjstate, ExprContext *econtext);
|
Add parallel-aware hash joins.
Introduce parallel-aware hash joins that appear in EXPLAIN plans as Parallel
Hash Join with Parallel Hash. While hash joins could already appear in
parallel queries, they were previously always parallel-oblivious and had a
partial subplan only on the outer side, meaning that the work of the inner
subplan was duplicated in every worker.
After this commit, the planner will consider using a partial subplan on the
inner side too, using the Parallel Hash node to divide the work over the
available CPU cores and combine its results in shared memory. If the join
needs to be split into multiple batches in order to respect work_mem, then
workers process different batches as much as possible and then work together
on the remaining batches.
The advantages of a parallel-aware hash join over a parallel-oblivious hash
join used in a parallel query are that it:
* avoids wasting memory on duplicated hash tables
* avoids wasting disk space on duplicated batch files
* divides the work of building the hash table over the CPUs
One disadvantage is that there is some communication between the participating
CPUs which might outweigh the benefits of parallelism in the case of small
hash tables. This is avoided by the planner's existing reluctance to supply
partial plans for small scans, but it may be necessary to estimate
synchronization costs in future if that situation changes. Another is that
outer batch 0 must be written to disk if multiple batches are required.
A potential future advantage of parallel-aware hash joins is that right and
full outer joins could be supported, since there is a single set of matched
bits for each hashtable, but that is not yet implemented.
A new GUC enable_parallel_hash is defined to control the feature, defaulting
to on.
Author: Thomas Munro
Reviewed-By: Andres Freund, Robert Haas
Tested-By: Rafia Sabih, Prabhat Sahu
Discussion:
https://postgr.es/m/CAEepm=2W=cOkiZxcg6qiFQP-dHUe09aqTrEMM7yJDrHMhDv_RA@mail.gmail.com
https://postgr.es/m/CAEepm=37HKyJ4U6XOLi=JgfSHM3o6B-GaeO-6hkOmneTDkH+Uw@mail.gmail.com
2017-12-21 02:39:21 -05:00
|
|
|
extern bool ExecParallelScanHashBucket(HashJoinState *hjstate, ExprContext *econtext);
|
2010-12-30 20:24:55 -05:00
|
|
|
extern void ExecPrepHashTableForUnmatched(HashJoinState *hjstate);
|
Parallel Hash Full Join.
Full and right outer joins were not supported in the initial
implementation of Parallel Hash Join because of deadlock hazards (see
discussion). Therefore FULL JOIN inhibited parallelism, as the other
join strategies can't do that in parallel either.
Add a new PHJ phase PHJ_BATCH_SCAN that scans for unmatched tuples on
the inner side of one batch's hash table. For now, sidestep the
deadlock problem by terminating parallelism there. The last process to
arrive at that phase emits the unmatched tuples, while others detach and
are free to go and work on other batches, if there are any, but
otherwise they finish the join early.
That unfairness is considered acceptable for now, because it's better
than no parallelism at all. The build and probe phases are run in
parallel, and the new scan-for-unmatched phase, while serial, is usually
applied to the smaller of the two relations and is either limited by
some multiple of work_mem, or it's too big and is partitioned into
batches and then the situation is improved by batch-level parallelism.
Author: Melanie Plageman <melanieplageman@gmail.com>
Author: Thomas Munro <thomas.munro@gmail.com>
Reviewed-by: Thomas Munro <thomas.munro@gmail.com>
Discussion: https://postgr.es/m/CA%2BhUKG%2BA6ftXPz4oe92%2Bx8Er%2BxpGZqto70-Q_ERwRaSyA%3DafNg%40mail.gmail.com
2023-03-30 18:01:51 -04:00
|
|
|
extern bool ExecParallelPrepHashTableForUnmatched(HashJoinState *hjstate);
|
2010-12-30 20:24:55 -05:00
|
|
|
extern bool ExecScanHashTableForUnmatched(HashJoinState *hjstate,
|
2019-05-22 13:04:48 -04:00
|
|
|
ExprContext *econtext);
|
Parallel Hash Full Join.
Full and right outer joins were not supported in the initial
implementation of Parallel Hash Join because of deadlock hazards (see
discussion). Therefore FULL JOIN inhibited parallelism, as the other
join strategies can't do that in parallel either.
Add a new PHJ phase PHJ_BATCH_SCAN that scans for unmatched tuples on
the inner side of one batch's hash table. For now, sidestep the
deadlock problem by terminating parallelism there. The last process to
arrive at that phase emits the unmatched tuples, while others detach and
are free to go and work on other batches, if there are any, but
otherwise they finish the join early.
That unfairness is considered acceptable for now, because it's better
than no parallelism at all. The build and probe phases are run in
parallel, and the new scan-for-unmatched phase, while serial, is usually
applied to the smaller of the two relations and is either limited by
some multiple of work_mem, or it's too big and is partitioned into
batches and then the situation is improved by batch-level parallelism.
Author: Melanie Plageman <melanieplageman@gmail.com>
Author: Thomas Munro <thomas.munro@gmail.com>
Reviewed-by: Thomas Munro <thomas.munro@gmail.com>
Discussion: https://postgr.es/m/CA%2BhUKG%2BA6ftXPz4oe92%2Bx8Er%2BxpGZqto70-Q_ERwRaSyA%3DafNg%40mail.gmail.com
2023-03-30 18:01:51 -04:00
|
|
|
extern bool ExecParallelScanHashTableForUnmatched(HashJoinState *hjstate,
|
|
|
|
|
ExprContext *econtext);
|
2005-03-06 17:15:05 -05:00
|
|
|
extern void ExecHashTableReset(HashJoinTable hashtable);
|
2010-12-30 20:24:55 -05:00
|
|
|
extern void ExecHashTableResetMatchFlags(HashJoinTable hashtable);
|
2009-03-20 20:04:40 -04:00
|
|
|
extern void ExecChooseHashTableSize(double ntuples, int tupwidth, bool useskew,
|
2020-07-29 17:14:58 -04:00
|
|
|
bool try_combined_hash_mem,
|
2019-05-22 13:04:48 -04:00
|
|
|
int parallel_workers,
|
|
|
|
|
size_t *space_allowed,
|
|
|
|
|
int *numbuckets,
|
|
|
|
|
int *numbatches,
|
|
|
|
|
int *num_skew_mcvs);
|
2009-03-20 20:04:40 -04:00
|
|
|
extern int ExecHashGetSkewBucket(HashJoinTable hashtable, uint32 hashvalue);
|
2017-12-05 13:55:56 -05:00
|
|
|
extern void ExecHashEstimate(HashState *node, ParallelContext *pcxt);
|
|
|
|
|
extern void ExecHashInitializeDSM(HashState *node, ParallelContext *pcxt);
|
|
|
|
|
extern void ExecHashInitializeWorker(HashState *node, ParallelWorkerContext *pwcxt);
|
|
|
|
|
extern void ExecHashRetrieveInstrumentation(HashState *node);
|
|
|
|
|
extern void ExecShutdownHash(HashState *node);
|
2020-04-11 12:39:19 -04:00
|
|
|
extern void ExecHashAccumInstrumentation(HashInstrumentation *instrument,
|
|
|
|
|
HashJoinTable hashtable);
|
2001-10-28 01:26:15 -05:00
|
|
|
|
Phase 2 of pgindent updates.
Change pg_bsd_indent to follow upstream rules for placement of comments
to the right of code, and remove pgindent hack that caused comments
following #endif to not obey the general rule.
Commit e3860ffa4dd0dad0dd9eea4be9cc1412373a8c89 wasn't actually using
the published version of pg_bsd_indent, but a hacked-up version that
tried to minimize the amount of movement of comments to the right of
code. The situation of interest is where such a comment has to be
moved to the right of its default placement at column 33 because there's
code there. BSD indent has always moved right in units of tab stops
in such cases --- but in the previous incarnation, indent was working
in 8-space tab stops, while now it knows we use 4-space tabs. So the
net result is that in about half the cases, such comments are placed
one tab stop left of before. This is better all around: it leaves
more room on the line for comment text, and it means that in such
cases the comment uniformly starts at the next 4-space tab stop after
the code, rather than sometimes one and sometimes two tabs after.
Also, ensure that comments following #endif are indented the same
as comments following other preprocessor commands such as #else.
That inconsistency turns out to have been self-inflicted damage
from a poorly-thought-through post-indent "fixup" in pgindent.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 15:18:54 -04:00
|
|
|
#endif /* NODEHASH_H */
|
