Commit graph

5259 commits

Author SHA1 Message Date
Peter Eisentraut
5b75a4f826 pgbench: Report errors during run better
When an error occurs during a benchmark run, exit with a nonzero exit
code and write a message at the end.  Previously, it would just print
the error message when it happened but then proceed to print the run
summary normally and exit with status 0.  To still allow
distinguishing setup from run-time errors, we use exit status 2 for
the new state, whereas existing errors during pgbench initialization
use exit status 1.

Reviewed-by: Fabien COELHO <coelho@cri.ensmp.fr>
2018-10-15 10:34:35 +02:00
Michael Paquier
b34e84f160 Add TAP tests for pg_verify_checksums
All options available in the utility get coverage:
- Tests with disabled page checksums.
- Tests with enabled test checksums.
- Emulation of corruption and broken checksums with a full scan and
single relfilenode scan.

This patch has been contributed mainly by Michael Banck and Magnus
Hagander with things presented on various threads, and I have gathered
all the contents into a single patch.

Author: Michael Banck, Magnus Hagander, Michael Paquier
Reviewed-by: Peter Eisentraut
Discussion: https://postgr.es/m/20181005012645.GE1629@paquier.xyz
2018-10-12 09:12:31 +09:00
Andres Freund
cda6a8d01d Remove deprecated abstime, reltime, tinterval datatypes.
These types have been deprecated for a *long* time.

Catversion bump, for obvious reasons.

Author: Andres Freund
Discussion:
    https://postgr.es/m/20181009192237.34wjp3nmw7oynmmr@alap3.anarazel.de
    https://postgr.es/m/20171213080506.cwjkpcz3bkk6yz2u@alap3.anarazel.de
    https://postgr.es/m/25615.1513115237@sss.pgh.pa.us
2018-10-11 11:59:15 -07:00
Magnus Hagander
a9da329be0 Fix speling error
Reported by Alexander Lakhin in bug #15423
2018-10-08 08:57:24 +02:00
Tom Lane
c87cb5f7a6 Allow btree comparison functions to return INT_MIN.
Historically we forbade datatype-specific comparison functions from
returning INT_MIN, so that it would be safe to invert the sort order
just by negating the comparison result.  However, this was never
really safe for comparison functions that directly return the result
of memcmp(), strcmp(), etc, as POSIX doesn't place any such restriction
on those library functions.  Buildfarm results show that at least on
recent Linux on s390x, memcmp() actually does return INT_MIN sometimes,
causing sort failures.

The agreed-on answer is to remove this restriction and fix relevant
call sites to not make such an assumption; code such as "res = -res"
should be replaced by "INVERT_COMPARE_RESULT(res)".  The same is needed
in a few places that just directly negated the result of memcmp or
strcmp.

To help find places having this problem, I've also added a compile option
to nbtcompare.c that causes some of the commonly used comparators to
return INT_MIN/INT_MAX instead of their usual -1/+1.  It'd likely be
a good idea to have at least one buildfarm member running with
"-DSTRESS_SORT_INT_MIN".  That's far from a complete test of course,
but it should help to prevent fresh introductions of such bugs.

This is a longstanding portability hazard, so back-patch to all supported
branches.

Discussion: https://postgr.es/m/20180928185215.ffoq2xrq5d3pafna@alap3.anarazel.de
2018-10-05 16:01:29 -04:00
Andres Freund
92a0342a90 Correct overflow handling in pgbench.
This patch attempts, although it's quite possible there are a few
holes, to properly detect and reported signed integer overflows in
pgbench.

Author: Fabien Coelho
Reviewed-By: Andres Freund
Discussion: https://postgr.es/m/20171212052943.k2hlckfkeft3eiio@alap3.anarazel.de
2018-09-27 21:50:57 -07:00
Tom Lane
d6c55de1f9 Implement %m in src/port/snprintf.c, and teach elog.c to rely on that.
I started out with the idea that we needed to detect use of %m format specs
in contexts other than elog/ereport calls, because we couldn't rely on that
working in *printf calls.  But a better answer is to fix things so that it
does work.  Now that we're using snprintf.c all the time, we can implement
%m in that and we've fixed the problem.

This requires also adjusting our various printf-wrapping functions so that
they ensure "errno" is preserved when they call snprintf.c.

Remove elog.c's handmade implementation of %m, and let it rely on
snprintf to support the feature.  That should provide some performance
gain, though I've not attempted to measure it.

There are a lot of places where we could now simplify 'printf("%s",
strerror(errno))' into 'printf("%m")', but I'm not in any big hurry
to make that happen.

Patch by me, reviewed by Michael Paquier

Discussion: https://postgr.es/m/2975.1526862605@sss.pgh.pa.us
2018-09-26 13:31:56 -04:00
Tom Lane
5b7e036707 Avoid unnecessary precision loss for pgbench's --rate target.
It's fairly silly to truncate the throttle_delay to integer when the only
math we ever do with it requires converting back to double.  Furthermore,
given that people are starting to complain about restrictions like only
supporting 1K client connections, I don't think we're very far away from
situations where the precision loss matters.  As the code stood, for
example, there's no difference between --rate 100001 and --rate 111111;
both get converted to throttle_delay = 9.  Somebody trying to run 100
threads and have each one dispatch around 1K TPS would find this lack of
precision rather surprising, especially since the required per-thread
delays are around 1ms, well within the timing precision of modern systems.
2018-09-25 11:09:18 -04:00
Michael Paquier
08c9917e24 Ignore publication tables when --no-publications is used
96e1cb4 has added support for --no-publications in pg_dump, pg_dumpall
and pg_restore, but forgot the fact that publication tables also need to
be ignored when this option is used.

Author: Gilles Darold
Reviewed-by: Michael Paquier
Discussion: https://postgr.es/m/3f48e812-b0fa-388e-2043-9a176bdee27e@dalibo.com
Backpatch-through: 10, where publications have been added.
2018-09-25 11:03:56 +09:00
Tom Lane
fd582317e1 Sync our Snowball stemmer dictionaries with current upstream.
We haven't touched these since text search functionality landed in core
in 2007 :-(.  While the upstream project isn't a beehive of activity,
they do make additions and bug fixes from time to time.  Update our
copies of these files.

Also update our documentation about how to keep things in sync, since
they're not making distribution tarballs these days.  Fortunately,
their source code turns out to be a breeze to build.

Notable changes:

* The non-UTF8 version of the hungarian stemmer now works in LATIN2
not LATIN1.

* New stemmers have appeared for arabic, indonesian, irish, lithuanian,
nepali, and tamil.  These all work in UTF8, and the indonesian and
irish ones also work in LATIN1.

(There are some new stemmers that I did not incorporate, mainly because
their names don't match the underlying languages, suggesting that they're
not to be considered mainstream.)

Worth noting: the upstream Nepali dictionary was contributed by
Arthur Zakirov.

initdb forced because the contents of snowball_create.sql have
changed.

Still TODO: see about updating the stopword lists.

Arthur Zakirov, minor mods and doc work by me

Discussion: https://postgr.es/m/20180626122025.GA12647@zakirov.localdomain
Discussion: https://postgr.es/m/20180219140849.GA9050@zakirov.localdomain
2018-09-24 17:29:38 -04:00
Tom Lane
60e612b602 Use ppoll(2), if available, to wait for input in pgbench.
Previously, pgbench always used select(2) for this purpose, but that's
problematic for very high client counts, because select() can't deal
with file descriptor numbers larger than FD_SETSIZE.  It's pretty common
for that to be only 1024 or so, whereas modern OSes can allow many more
open files than that.  Using poll(2) would surmount that problem, but it
creates another one: poll()'s timeout resolution is only 1ms, which is
poor enough to cause problems with --rate specifications approaching or
exceeding 1K TPS.

On platforms that have ppoll(2), which includes Linux and recent
FreeBSD, we can use that to avoid the FD_SETSIZE problem without any
loss of timeout resolution.  Hence, add configure logic to test for
ppoll(), and use it if available.

This patch introduces an abstraction layer into pgbench that could
be extended to support other kernel event-wait APIs such as kevents.
But actually adding such support is a matter for some future patch.

Doug Rady, reviewed by Robert Haas and Fabien Coelho, and whacked around
a good bit more by me

Discussion: https://postgr.es/m/23D017C9-81B7-484D-8490-FD94DEC4DF59@amazon.com
2018-09-24 14:40:58 -04:00
Noah Misch
d18f6674bd Initialize random() in bootstrap/stand-alone postgres and in initdb.
This removes a difference between the standard IsUnderPostmaster
execution environment and that of --boot and --single.  In a stand-alone
backend, "SELECT random()" always started at the same seed.

On a system capable of using posix shared memory, initdb could still
conclude "selecting dynamic shared memory implementation ... sysv".
Crashed --boot or --single postgres processes orphaned shared memory
objects having names that collided with the not-actually-random names
that initdb probed.  The sysv fallback appeared after ten crashes of
--boot or --single postgres.  Since --boot and --single are rare in
production use, systems used for PostgreSQL development are the
principal candidate to notice this symptom.

Back-patch to 9.3 (all supported versions).  PostgreSQL 9.4 introduced
dynamic shared memory, but 9.3 does share the "SELECT random()" problem.

Reviewed by Tom Lane and Kyotaro HORIGUCHI.

Discussion: https://postgr.es/m/20180915221546.GA3159382@rfd.leadboat.com
2018-09-23 22:56:39 -07:00
Tom Lane
4f3b38fe2b Get rid of explicit argument-count markings in tab-complete.c.
This replaces the "TailMatchesN" macros with just "TailMatches",
and likewise "HeadMatchesN" becomes "HeadMatches" and "MatchesN"
becomes "Matches".  The various COMPLETE_WITH_LISTn macros are
reduced to COMPLETE_WITH, and the single-item COMPLETE_WITH_CONST
also gets folded into that.  This eliminates a lot of minor
annoyance in writing tab-completion rules.  Usefully, the compiled
code also gets a bit smaller (10% or so, on my machine).

The implementation depends on variadic macros, so we couldn't have
done this before we required C99.

Andres Freund and Thomas Munro; some cosmetic cleanup by me.

Discussion: https://postgr.es/m/d8jo9djvm7h.fsf@dalvik.ping.uio.no
2018-09-21 20:50:41 -04:00
Tom Lane
e8fe426baa Fix bogus tab-completion rule for CREATE PUBLICATION.
You can't use "FOR TABLE" as a single Matches argument, because readline
will consider that input to be two words not one.  It's necessary to make
the pattern contain two arguments.

The case accidentally worked anyway because the words_after_create
test fired ... but only for the first such table name.

Noted by Edmund Horner, though this isn't exactly his proposed fix.
Backpatch to v10 where the faulty code came in.

Discussion: https://postgr.es/m/CAMyN-kDe=gBmHgxWwUUaXuwK+p+7g1vChR7foPHRDLE592nJPQ@mail.gmail.com
2018-09-21 15:58:37 -04:00
Tom Lane
121213d9d8 Improve tab completion for ANALYZE, EXPLAIN, and VACUUM.
Previously, we made no attempt to provide tab completion in these
statements' optional parenthesized options lists.  This patch teaches
psql to do so.

To prevent the option completions from being offered after we've already
seen a complete parenthesized option list, it's necessary to improve
word_matches() so that it allows a wildcard '*' in the middle of an
alternative, not only at the end as formerly.  That requires only a
little more code than before, and it allows us to test for "incomplete
parenthesized options" with a test like

    else if (HeadMatches2("EXPLAIN", "(*") &&
             !HeadMatches2("EXPLAIN", "(*)"))

In addition, add some logic to offer column names in the context of
"ANALYZE tablename ( ...", and likewise for VACUUM.  This isn't real
complete; it won't offer column names again after a comma.  But it's
better than before, and it doesn't take much code.

Justin Pryzby, reviewed at various times by Álvaro Herrera, Arthur
Zakirov, and Edmund Horner; some additional fixups by me

Discussion: https://postgr.es/m/20180529000623.GA21896@telsasoft.com
2018-09-21 15:22:26 -04:00
Tom Lane
e3b7a6d165 Rationalize Query_for_list_of_[relations] query names in tab-complete.c.
The previous convention was to use names based on the set of relkinds being
selected for, which was not at all helpful for maintenance, especially
since people had been quite inconsistent about whether to change the names
when they changed the relkinds being selected for.  Instead, use names
based on the functionality we need the relation to have, following the
model established by Query_for_list_of_updatables.

While at it, sort the list of Query constants a bit better; it had the
distinct air of code-assembled-by-dartboard before.

Discussion: https://postgr.es/m/14830.1537481254@sss.pgh.pa.us
2018-09-21 12:41:00 -04:00
Michael Paquier
925673f27b Remove special handling for open() in initdb for Windows
40cfe86 enforces the translation mode to text for all frontends, so this
special handling in initdb is not needed anymore.
2018-09-21 06:41:44 +09:00
Tom Lane
c9a8a401f1 Fix psql's tab completion for TABLE.
This should offer the same relation types that SELECT ... FROM would.
You can't select from an index for instance, so offering it here is
unhelpful.  Noted while testing ilmari's recent patch.
2018-09-20 17:21:14 -04:00
Tom Lane
a7c4dad1a7 Fix psql's tab completion for ALTER DATABASE ... SET TABLESPACE.
We have the infrastructure to offer a list of tablespace names, but
it wasn't being used here; instead you got "FROM", "CURRENT", and "TO"
which aren't actually legal in this syntax.

Dagfinn Ilmari Mannsåker, reviewed by Arthur Zakirov

Discussion: https://postgr.es/m/d8jo9djvm7h.fsf@dalvik.ping.uio.no
2018-09-20 17:16:06 -04:00
Tom Lane
789ba5029a Remove dead code from pop_next_work_item().
The pref_non_data heuristic has been dead code for nearly ten years,
and as far as I can tell was dead code even when it was first committed.
I'm tired of silencing Coverity complaints about it, so get rid of it.
If anyone is ever interested in pursuing the concept, they can get the
code out of our git history.
2018-09-17 12:43:07 -04:00
Tom Lane
db37ab2c60 Fix pgbench lexer's "continuation" rule to cope with Windows newlines.
Our general practice in frontend code is to accept input with either
Unix-style newlines (\n) or DOS-style (\r\n).  pgbench was mostly down
with that, but its rule for line continuations (backslash-newline) was
not.  This had been masked on Windows buildfarm machines before commit
0ba06e0bf by use of Windows text mode to read files.  We could have fixed
it by forcing text mode again, but it's better to fix the parsing code
so that Windows-style text files on Unix systems don't cause problems.

Back-patch to v10 where pgbench grew line continuations.

Discussion: https://postgr.es/m/17194.1537191697@sss.pgh.pa.us
2018-09-17 12:11:43 -04:00
Tom Lane
548e50976c Improve parallel scheduling logic in pg_dump/pg_restore.
Previously, the way this worked was that a parallel pg_dump would
re-order the TABLE_DATA items in the dump's TOC into decreasing size
order, and separately re-order (some of) the INDEX items into decreasing
size order.  Then pg_dump would dump the items in that order.  Later,
parallel pg_restore just followed the TOC order.  This method had lots
of deficiencies:

* TOC ordering randomly differed between parallel and non-parallel
dumps, and was hard to predict in the former case, causing problems
for building stable pg_dump test cases.

* Parallel restore only followed a well-chosen order if the dump had
been done in parallel; in particular, this never happened for restore
from custom-format dumps.

* The best order for restore isn't necessarily the same as for dump,
and it's not really static either because of locking considerations.

* TABLE_DATA and INDEX items aren't the only things that might take a lot
of work during restore.  Scheduling was particularly stupid for the BLOBS
item, which might require lots of work during dump as well as restore,
but was left to the end in either case.

This patch removes the logic that changed the TOC order, fixing the
test instability problem.  Instead, we sort the parallelizable items
just before processing them during a parallel dump.  Independently
of that, parallel restore prioritizes the ready-to-execute tasks
based on the size of the underlying table.  In the case of dependent
tasks such as index, constraint, or foreign key creation, the largest
relevant table is used as the metric for estimating the task length.
(This is pretty crude, but it should be enough to avoid the case we
want to avoid, which is ending the run with just a few large tasks
such that we can't make use of all N workers.)

Patch by me, responding to a complaint from Peter Eisentraut,
who also reviewed the patch.

Discussion: https://postgr.es/m/5137fe12-d0a2-4971-61b6-eb4e7e8875f8@2ndquadrant.com
2018-09-14 17:31:51 -04:00
Michael Paquier
0ba06e0bfb Allow concurrent-safe open() and fopen() in frontend code for Windows
PostgreSQL uses a custom wrapper for open() and fopen() which is
concurrent-safe, allowing multiple processes to open and work on the
same file.  This has a couple of advantages:
- pg_test_fsync does not handle O_DSYNC correctly otherwise, leading to
false claims that disks are unsafe.
- TAP tests can run into race conditions when a postmaster and pg_ctl
open postmaster.pid, fixing some random failures in the buildfam.

pg_upgrade is one frontend tool using workarounds to bypass file locking
issues with the log files it generates, however the interactions with
pg_ctl are proving to be tedious to get rid of, so this is left for
later.

Author: Laurenz Albe
Reviewed-by: Michael Paquier, Kuntal Ghosh
Discussion: https://postgr.es/m/1527846213.2475.31.camel@cybertec.at
Discussion: https://postgr.es/m/16922.1520722108@sss.pgh.pa.us
2018-09-14 10:04:14 +09:00
Peter Eisentraut
f48fa2bc8b Message style improvements
Fix one untranslatable string concatenation in pg_rewind.

Fix one message in pg_verify_checksums to use a style use elsewhere
and avoid plural issues.

Fix one gratuitous abbreviation in psql.
2018-09-13 23:35:43 +02:00
Tom Lane
23bd3cec6e Attempt to identify system timezone by reading /etc/localtime symlink.
On many modern platforms, /etc/localtime is a symlink to a file within the
IANA database.  Reading the symlink lets us find out the name of the system
timezone directly, without going through the brute-force search embodied in
scan_available_timezones().  This shortens the runtime of initdb by some
tens of ms, which is helpful for the buildfarm, and it also allows us to
reliably select the same zone name the system was actually configured for,
rather than possibly choosing one of IANA's many zone aliases.  (For
example, in a system configured for "Asia/Tokyo", the brute-force search
would not choose that name but its alias "Japan", on the grounds of the
latter string being shorter.  More surprisingly, "Navajo" is preferred
to either "America/Denver" or "US/Mountain", as seen in an old complaint
from Josh Berkus.)

If /etc/localtime doesn't exist, or isn't a symlink, or we can't make
sense of its contents, or the contents match a zone we know but that
zone doesn't match the observed behavior of localtime(), fall back to
the brute-force search.

Also, tweak initdb so that it prints the zone name it selected.

In passing, replace the last few references to the "Olson" database in
code comments with "IANA", as that's been our preferred term since
commit b2cbced9e.

Patch by me, per a suggestion from Robert Haas; review by Michael Paquier

Discussion: https://postgr.es/m/7408.1525812528@sss.pgh.pa.us
2018-09-13 12:36:21 -04:00
Tom Lane
7046d30246 Minor fixes for psql tab completion.
* Include partitioned tables in what's offered after ANALYZE.

* Include toast_tuple_target in what's offered after ALTER TABLE ... SET|RESET.

* Include HASH in what's offered after PARTITION BY.

This is extracted from a larger patch; these bits seem like
uncontroversial bug fixes for v11 features, so back-patch them into v11.

Justin Pryzby

Discussion: https://postgr.es/m/20180529000623.GA21896@telsasoft.com
2018-09-12 15:25:12 -04:00
Peter Eisentraut
98afa68d93 Use C99 designated initializers for some structs
These are just a few particularly egregious cases that were hard to read
and write, and error prone because of many similar adjacent types.

Discussion: https://www.postgresql.org/message-id/flat/4c9f01be-9245-2148-b569-61a8562ef190%402ndquadrant.com
2018-09-07 11:40:03 +02:00
Tom Lane
44cac93464 Avoid using potentially-under-aligned page buffers.
There's a project policy against using plain "char buf[BLCKSZ]" local
or static variables as page buffers; preferred style is to palloc or
malloc each buffer to ensure it is MAXALIGN'd.  However, that policy's
been ignored in an increasing number of places.  We've apparently got
away with it so far, probably because (a) relatively few people use
platforms on which misalignment causes core dumps and/or (b) the
variables chance to be sufficiently aligned anyway.  But this is not
something to rely on.  Moreover, even if we don't get a core dump,
we might be paying a lot of cycles for misaligned accesses.

To fix, invent new union types PGAlignedBlock and PGAlignedXLogBlock
that the compiler must allocate with sufficient alignment, and use
those in place of plain char arrays.

I used these types even for variables where there's no risk of a
misaligned access, since ensuring proper alignment should make
kernel data transfers faster.  I also changed some places where
we had been palloc'ing short-lived buffers, for coding style
uniformity and to save palloc/pfree overhead.

Since this seems to be a live portability hazard (despite the lack
of field reports), back-patch to all supported versions.

Patch by me; thanks to Michael Paquier for review.

Discussion: https://postgr.es/m/1535618100.1286.3.camel@credativ.de
2018-09-01 15:27:17 -04:00
Alexander Korotkov
ec74369931 Implement "pg_ctl logrotate" command
Currently there are two ways to trigger log rotation in logging collector
process: call pg_rotate_logfile() SQL-function or send SIGUSR1 signal directly
to logging collector process.  However, it's nice to have more suitable way
for external tools to do that, which wouldn't require SQL connection or
knowledge of logging collector pid.  This commit implements triggering log
rotation by "pg_ctl logrotate" command.

Discussion: https://postgr.es/m/20180416.115435.28153375.horiguchi.kyotaro%40lab.ntt.co.jp
Author: Kyotaro Horiguchi, Alexander Kuzmenkov, Alexander Korotkov
2018-09-01 19:46:49 +03:00
Tom Lane
115bf1e79a Fix psql's \dC command to annotate I/O conversion casts as such.
A cast declared WITH INOUT was described as '(binary coercible)',
which seems pretty inaccurate; let's print '(with inout)' instead.
Per complaint from Jean-Pierre Pelletier.

This definitely seems like a bug fix, but given that it's been wrong
since 8.4 and nobody complained before, I'm hesitant to back-patch a
behavior change into stable branches.  It doesn't seem too late for
v11 though.

Discussion: https://postgr.es/m/5b887023.1c69fb81.ff96e.6a1d@mx.google.com
2018-08-31 16:45:33 -04:00
Tom Lane
d9c366f9e8 Code review for pg_verify_checksums.c.
Use postgres_fe.h, since this is frontend code.  Pretend that we've heard
of project style guidelines for, eg, #include order.  Use BlockNumber not
int arithmetic for block numbers, to avoid misbehavior with relations
exceeding 2^31 blocks.  Avoid an unnecessary strict-aliasing warning
(per report from Michael Banck).  Const-ify assorted stuff.  Avoid
scribbling on the output of readdir() -- perhaps that's safe in practice,
but POSIX forbids it, and this code has so far earned exactly zero
credibility portability-wise.  Editorialize on an ambiguously-worded
message.

I did not touch the problem of the "buf" local variable being possibly
insufficiently aligned; that's not specific to this code, and seems like
it should be fixed as part of a different, larger patch.

Discussion: https://postgr.es/m/1535618100.1286.3.camel@credativ.de
2018-08-31 13:42:28 -04:00
Amit Kapila
bb60f2cb6e Fix pg_verify_checksums on Windows.
To verify the checksums, we open the file in text mode which doesn't work
on Windows as WIN32 treats Control-Z as EOF in files opened in text mode.
This leads to "short read of block .." error in some cases.

Fix it by opening the files in the binary mode.

Author: Amit Kapila
Reviewed-by: Magnus Hagander
Backpatch-through: 11
Discussion: https://postgr.es/m/CAA4eK1+LOnzod+h85FGmyjWzXKy-XV1FYwEyP-Tky2WpD5cxwA@mail.gmail.com
2018-08-31 15:25:42 +05:30
Peter Eisentraut
7061e03319 Add semicolons to end of internally run queries
This ensures that the --echo output of various tools (under scripts) is
valid multiline SQL.

Author: Tatsuro Yamada <yamada.tatsuro@lab.ntt.co.jp>
2018-08-30 19:23:22 +02:00
Peter Eisentraut
daa9fe8a52 pg_dump: Reorganize getTableAttrs()
Instead of repeating the almost same large query in each version branch,
use one query and add a few columns to the SELECT list depending on the
version.  This saves a lot of duplication.

Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
2018-08-30 18:09:44 +02:00
Alvaro Herrera
a846e6d023 pg_verify_checksums: rename -d to --verbose
Using -d is odd, because we normally reserve that for a database
argument, so rename it to -v and add long version --verbose.

Also, reduce it to emit one line per file checked rather than one line
per block.

Per a complaint from Michael Banck.

Author: Yugo Nagata <nagata@sraoss.co.jp>
Reviewed-by: Michael Banck <michael.banck@credativ.de>
Discussion: https://postgr.es/m/20180827113411.GA22768@nighthawk.caipicrew.dd-dns.de
2018-08-30 06:35:55 -03:00
Tom Lane
e0a0cc28d0 Make pg_restore's identify_locking_dependencies() more bulletproof.
This function had a blacklist of dump object types that it believed
needed exclusive lock ... but we hadn't maintained that, so that it
was missing ROW SECURITY, POLICY, and INDEX ATTACH items, all of
which need (or should be treated as needing) exclusive lock.

Since the same oversight seems likely in future, let's reverse the
sense of the test so that the code has a whitelist of safe object
types; better to wrongly assume a command can't be run in parallel
than the opposite.  Currently the only POST_DATA object type that's
safe is CREATE INDEX ... and that list hasn't changed in a long time.

Back-patch to 9.5 where RLS came in.

Discussion: https://postgr.es/m/11450.1535483506@sss.pgh.pa.us
2018-08-28 19:46:59 -04:00
Tom Lane
8cff4f5348 Code review for pg_dump's handling of ALTER INDEX ATTACH PARTITION.
Ensure the TOC entry is marked with the correct schema, so that its
name is as unique as the index's is.

Fix the dependencies: we want dependencies from this TOC entry to the
two indexes it depends on, and we don't care (at least not for this
purpose) what order the indexes are created in.  Also, add dependencies
on the indexes' underlying tables.  Those might seem pointless given
the index dependencies, but they are helpful to cue parallel restore
to avoid running the ATTACH PARTITION in parallel with other DDL on
the same tables.

Discussion: https://postgr.es/m/10817.1535494963@sss.pgh.pa.us
2018-08-28 19:33:04 -04:00
Peter Eisentraut
3e2ceb231e pg_verify_checksums: Message style improvements and NLS support
The source code was already set up for NLS support, so just a nls.mk
file needed to be added.  Also, fix the old problem of putting the int64
format specifier right into the string, which breaks NLS.
2018-08-28 11:49:11 +02:00
Tom Lane
cbdca00bef Fix missing dependency for pg_dump's ENABLE ROW LEVEL SECURITY items.
The archive should show a dependency on the item's table, but it failed
to include one.  This could cause failures in parallel restore due to
emitting ALTER TABLE ... ENABLE ROW LEVEL SECURITY before restoring
the table's data.  In practice the odds of a problem seem low, since
you would typically need to have set FORCE ROW LEVEL SECURITY as well,
and you'd also need a very high --jobs count to have any chance of this
happening.  That probably explains the lack of field reports.

Still, it's a bug, so back-patch to 9.5 where RLS was introduced.

Discussion: https://postgr.es/m/19784.1535390902@sss.pgh.pa.us
2018-08-27 15:11:12 -04:00
Peter Eisentraut
2d41d914ab Copy-editing of pg_verify_checksums help and ref page
Reformat synopsis, put options into better order, make the desciption
line a bit shorter, and put more details into the description.
2018-08-23 20:32:56 +02:00
Peter Eisentraut
d12782898e Change PROCEDURE to FUNCTION in CREATE OPERATOR syntax
Since procedures are now a different thing from functions, change the
CREATE OPERATOR syntax to use FUNCTION in the clause that specifies the
function.  PROCEDURE is still accepted for compatibility.

Reported-by: Peter Geoghegan <pg@bowt.ie>
Reviewed-by: Jonathan S. Katz <jonathan.katz@excoventures.com>
2018-08-22 14:44:49 +02:00
Peter Eisentraut
b19495772e doc: Update uses of the word "procedure"
Historically, the term procedure was used as a synonym for function in
Postgres/PostgreSQL.  Now we have procedures as separate objects from
functions, so we need to clean up the documentation to not mix those
terms.

In particular, mentions of "trigger procedures" are changed to "trigger
functions", and access method "support procedures" are changed to
"support functions".  (The latter already used FUNCTION in the SQL
syntax anyway.)  Also, the terminology in the SPI chapter has been
cleaned up.

A few tests, examples, and code comments are also adjusted to be
consistent with documentation changes, but not everything.

Reported-by: Peter Geoghegan <pg@bowt.ie>
Reviewed-by: Jonathan S. Katz <jonathan.katz@excoventures.com>
2018-08-22 14:44:49 +02:00
Michael Paquier
b965f26171 Do not dump identity sequences with excluded parent table
This commit prevents a crash of pg_dump caused by the exclusion of a
table which has identity columns, as the table would be correctly
excluded but not its identity sequence.  In order to fix that, identity
sequences are excluded if the parent table is defined as such.  Knowing
about such sequences has no meaning without their parent table anyway.

Reported-by: Andy Abelisto
Author: David Rowley
Reviewed-by: Peter Eisentraut, Michael Paquier
Discussion: https://postgr.es/m/153479393218.1316.8472285660264976457@wrigleys.postgresql.org
Backpatch-through: 10
2018-08-22 14:21:49 +09:00
Michael Paquier
72be8c29a1 Fix set of NLS translation issues
While monitoring the code, a couple of issues related to string
translation has showed up:
- Some routines for auto-updatable views return an error string, which
sometimes missed the shot.  A comment regarding string translation is
added for each routine to help with future features.
- GSSAPI authentication missed two translations.
- vacuumdb handles non-translated strings.
- GetConfigOptionByNum should translate strings.  This part is not
back-patched as after a minor upgrade this could be surprising for
users.

Reported-by: Kyotaro Horiguchi
Author: Kyotaro Horiguchi
Reviewed-by: Michael Paquier, Tom Lane
Discussion: https://postgr.es/m/20180810.152131.31921918.horiguchi.kyotaro@lab.ntt.co.jp
Backpatch-through: 9.3
2018-08-21 15:17:13 +09:00
Tom Lane
6771c932cf Ensure schema qualification in pg_restore DISABLE/ENABLE TRIGGER commands.
Previously, this code blindly followed the common coding pattern of
passing PQserverVersion(AH->connection) as the server-version parameter
of fmtQualifiedId.  That works as long as we have a connection; but in
pg_restore with text output, we don't.  Instead we got a zero from
PQserverVersion, which fmtQualifiedId interpreted as "server is too old to
have schemas", and so the name went unqualified.  That still accidentally
managed to work in many cases, which is probably why this ancient bug went
undetected for so long.  It only became obvious in the wake of the changes
to force dump/restore to execute with restricted search_path.

In HEAD/v11, let's deal with this by ripping out fmtQualifiedId's server-
version behavioral dependency, and just making it schema-qualify all the
time.  We no longer support pg_dump from servers old enough to need the
ability to omit schema name, let alone restoring to them.  (Also, the few
callers outside pg_dump already didn't work with pre-schema servers.)

In older branches, that's not an acceptable solution, so instead just
tweak the DISABLE/ENABLE TRIGGER logic to ensure it will schema-qualify
its output regardless of server version.

Per bug #15338 from Oleg somebody.  Back-patch to all supported branches.

Discussion: https://postgr.es/m/153452458706.1316.5328079417086507743@wrigleys.postgresql.org
2018-08-17 17:12:33 -04:00
Bruce Momjian
b94f7b5350 pg_upgrade: issue helpful error message for use on standbys
Commit 777e6ddf17 checked for a shut down
message from a standby and allowed it to continue.  This patch reports a
helpful error message in these cases, suggesting to use rsync as
documented.

Diagnosed-by: Martín Marqués

Discussion: https://postgr.es/m/CAPdiE1xYCow-reLjrhJ9DqrMu-ppNq0ChUUEvVdxhdjGRD5_eA@mail.gmail.com

Backpatch-through: 9.3
2018-08-17 10:25:48 -04:00
Bruce Momjian
777e6ddf17 pg_upgrade: fix shutdown check for standby servers
Commit 244142d32a only tested for the
pg_controldata output for primary servers, but standby servers have
different "Database cluster state" output, so check for that too.

Diagnosed-by: Michael Paquier

Discussion: https://postgr.es/m/20180810164240.GM13638@paquier.xyz

Backpatch-through: 9.3
2018-08-14 17:19:02 -04:00
Tom Lane
d4a900458e Cosmetic cleanups in initdb.c.
Commit bcbd94080 didn't bother to fix up a comment it had falsified.

Also, const-ify choose_dsm_implementation(), since what it's returning
is always a constant string; pickier compilers would warn about this.
2018-08-10 14:14:27 -04:00
Tom Lane
03838b8049 Fix pg_upgrade to handle event triggers in extensions correctly.
pg_dump with --binary-upgrade must emit ALTER EXTENSION ADD commands
for all objects that are members of extensions.  It forgot to do so for
event triggers, as per bug #15310 from Nick Barnes.  Back-patch to 9.3
where event triggers were introduced.

Haribabu Kommi

Discussion: https://postgr.es/m/153360083872.1395.4593932457718151600@wrigleys.postgresql.org
2018-08-07 15:43:48 -04:00
Tom Lane
5b5ed4756c Ensure pg_dump_sort.c sorts null vs non-null namespace consistently.
The original coding here (which is, I believe, my fault) supposed that
it didn't need to concern itself with the possibility that one object
of a given type-priority has a namespace while another doesn't.  But
that's not reliably true anymore, if it ever was; and if it does happen
then it's possible that DOTypeNameCompare returns self-inconsistent
comparison results.  That leads to unspecified behavior in qsort()
and a resultant weird output order from pg_dump.

This should end up being only a cosmetic problem, because any ordering
constraints that actually matter should be enforced by the later
dependency-based sort.  Still, it's a bug, so back-patch.

Report and fix by Jacob Champion, though I editorialized on his
patch to the extent of making NULL sort after non-NULL, for consistency
with our usual sorting definitions.

Discussion: https://postgr.es/m/CABAq_6Hw+V-Kj7PNfD5tgOaWT_-qaYkc+SRmJkPLeUjYXLdxwQ@mail.gmail.com
2018-08-07 13:13:42 -04:00