This centralizes the check for valid nonce lengths for AES-GCM.
While here, remove some duplicate checks for valid AES-GCM tag lengths
from ccp(4) and ccr(4).
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33194
(cherry picked from commit 6e17a2e00d)
Currently armv8crypto copies the scheme used in aesni(9), where payload
data and output buffers are allocated on the fly if the crypto buffer is
not virtually contiguous. This scheme is simple but incurs a lot of
overhead: for an encryption request with a separate output buffer we
have to
- allocate a temporary buffer to hold the payload
- copy input data into the buffer
- copy the encrypted payload to the output buffer
- zero the temporary buffer before freeing it
We have a handy crypto buffer cursor abstraction now, so reimplement the
armv8crypto routines using that instead of temporary buffers. This
introduces some extra complexity, but gallatin@ reports a 10% throughput
improvement with a KTLS workload without additional CPU usage. The
driver still allocates an AAD buffer for AES-GCM if necessary.
Reviewed by: jhb
Tested by: gallatin
Sponsored by: Ampere Computing LLC
Submitted by: Klara Inc.
(cherry picked from commit 26b08c5d21)
This is in preparation for using buffer cursors. No functional change
intended.
Reviewed by: jhb
Sponsored by: Ampere Computing LLC
Submitted by: Klara Inc.
(cherry picked from commit 0b3235ef74)
These ones were unambiguous cases where the Foundation was the only
listed copyright holder (in the associated license block).
Sponsored by: The FreeBSD Foundation
(cherry picked from commit 9feff969a0)
As with sha256 add support for accelerated sha512 support to libmd on
arm64. This depends on clang 13+ to build as this is the first release
with the needed intrinsics. Gcc should also support them, however from
a currently unknown release.
Reviewed by: cem
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33373
(cherry picked from commit 71bf1c4cc5)
This is the same as 4285655adb ("aesni: Avoid a potential
out-of-bounds load in AES_GCM_encrypt()") except for the decryption
path.
Reported by: Jenkins (KASAN job)
Reviewed by: cem
Sponsored by: The FreeBSD Foundation
(cherry picked from commit 4a61d8ef42)
GCC 9 doesn't define a SHA256_Transform symbol when the stub just wraps
SHA256_Transform_c resulting in an undefined symbol for
_libmd_SHA256_Transform in libmd.so.
Discussed with: andrew, jrtc27
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D31945
(cherry picked from commit a6765c4b44)
Summary:
When running on a CPU that supports the arm64 sha256 intrinsics use them
to improve perfromance of sha256 calculations.
With this changethe following improvement has been seen on an Apple M1
with FreeBS running under Parallels, with similar results on a
Neoverse-N1 r3p1.
x sha256.orig
+ sha256.arm64
+--------------------------------------------------------------------+
|++ x x|
|+++ xxx|
||A |A||
+--------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 5 3.41 3.5 3.46 3.458 0.042661458
+ 5 0.47 0.54 0.5 0.504 0.027018512
Difference at 95.0% confidence
-2.954 +/- 0.0520768
-85.4251% +/- 0.826831%
(Student's t, pooled s = 0.0357071)
Reviewed by: cem
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31284
(cherry picked from commit 69d8dc20be)
This is useful for WireGuard which uses a nonce of 8 bytes rather
than the 12 bytes used for IPsec and TLS.
Note that this also fixes a (should be) harmless bug in ossl(4) where
the counter was incorrectly treated as a 64-bit counter instead of a
32-bit counter in terms of wrapping when using a 12 byte nonce.
However, this required a single message (TLS record) longer than 64 *
(2^32 - 1) bytes (about 256 GB) to trigger.
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32122
(cherry picked from commit 42dcd39528)
Add a 'len' argument to the reinit hook in 'struct enc_xform' to
permit support for AEAD ciphers such as AES-CCM and Chacha20-Poly1305
which support different nonce lengths.
Reviewed by: markj
Sponsored by: Chelsio Communications, The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32105
(cherry picked from commit 1833d6042c)
(cherry picked from commit d586c978b9)
This is intended for use in KTLS transmit where each TLS record is
described by a single mbuf that is itself queued in the socket buffer.
Using the existing CRYPTO_BUF_MBUF would result in
bus_dmamap_load_crp() walking additional mbufs in the socket buffer
that are not relevant, but generating a S/G list that potentially
exceeds the limit of the tag (while also wasting CPU cycles).
Reviewed by: markj
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D30136
(cherry picked from commit 883a0196b6)
While here, use crypto_read_iv() in a few more places in ccr(4) that I
missed previously.
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D32104
(cherry picked from commit cb128893b9)
The loops for Chacha20 and Chacha20+Poly1305 which encrypted/decrypted
full blocks of data used the minimum of the input and output segment
lengths to determine the size of the next chunk ('todo') to pass to
Chacha20_ctr32(). However, the input and output segments could extend
past the end of the ciphertext region into the tag (e.g. if a "plain"
single mbuf contained an entire TLS record). If the length of the tag
plus the length of the last partial block together were at least as
large as a full Chacha20 block (64 bytes), then an extra block was
encrypted/decrypted overlapping with the tag. Fix this by also
capping the amount of data to encrypt/decrypt by the amount of
remaining data in the ciphertext region ('resid').
Reported by: gallatin
Reviewed by: cem, gallatin, markj
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D29517
(cherry picked from commit d2e076c37b)
This file inherits some boilerplate and structure from the analogous
file in aesni(4), aesni_wrap.c. Note the derivation and the copyright
holders of that file.
For example, the AES-XTS bits added in 4979620ece were ported from
aesni(4).
Requested by: jmg
Reviewed by: imp, gnn
(cherry picked from commit da45b46266)
Initialization of the XTS key schedule was accidentally dropped
when adding AES-GCM support so all-zero schedule was used instead.
This rendered previously created GELI partitions unusable.
This change restores proper XTS key schedule initialization.
Reported by: Peter Jeremy <peter@rulingia.com>
MFC after: immediately
(cherry picked from commit 748be78e60)
The missing newline mildly garbles boot-time messages and this can be
troublesome if you need those.
Reported by: Mike Karels (mike@karels.net)
Reviewed By: gonzo
(cherry picked from commit 530d38441d)
This makes it easier to refactor the GCM code to operate on
crypto_buffer_cursors rather than plain contiguous buffers, with the aim
of minimizing the amount of copying and zeroing done today.
No functional change intended.
Reviewed by: jhb
Sponsored by: Ampere Computing
Submitted by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D28500
(cherry picked from commit 7509b677b4)
- We were only hashing up to the first 16 bytes of the AAD.
- When computing the digest during decryption, handle the case where
len == trailer, i.e., len < AES_BLOCK_LEN, properly.
While here:
- trailer is always smaller than AES_BLOCK_LEN, so remove a pair of
unnecessary modulus operations.
- Replace some byte-by-byte loops with memcpy() and memset() calls.
In particular, zero the full block before copying a partial block into
it since we do that elsewhere and it means that the memset() length is
known at compile time.
Reviewed by: jhb
Sponsored by: Ampere Computing
Submitted by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D28501
(cherry picked from commit 0dc7076037)
Rather than depending on malloc() returning 16-byte aligned chunks,
allocate some extra pad bytes and ensure that key schedules are
appropriately aligned.
Reviewed by: kib
MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC (Netgate)
Differential Revision: https://reviews.freebsd.org/D28157
Similar to the message printed by aesni(4), let the user know if the
driver is unsupported by their CPU.
PR: 252543
Reported by: gbe
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
A straightforward(ish) port from aesni(4). This implementation does not
perform loop unrolling on the input blocks, so this is left as a future
performance improvement.
Submitted by: Greg V <greg AT unrelenting.technology>
Looks good: jhb, jmg
Tested by: mhorne
Differential Revision: https://reviews.freebsd.org/D21017
Follow-up to r353959 and r368070: do the same for other architectures.
arm32 already seems to use its own .fnstart/.fnend directives, which
appear to be ARM-specific variants of the same thing. Likewise, MIPS
uses .frame directives.
Reviewed by: arichardson
Differential Revision: https://reviews.freebsd.org/D27387
Enable in-kernel acceleration of SHA1 and SHA2 operations on arm64 by adding
support for the ossl(4) crypto driver. This uses OpenSSL's assembly routines
under the hood, which will detect and use SHA intrinsics if they are
supported by the CPU.
Reviewed by: jhb
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D27390
Make room for adding arm64 support to this driver by moving the
x86-specific feature parsing to a separate file.
Reviewed by: jhb
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D27388
OCF drivers in general should perform as many session parameter checks
as possible during probesession rather than when creating a new
session. I got this wrong for aesni(4) in r359374. In addition,
aesni(4) was performing the check for digest-only requests and failing
to create digest-only sessions as a result.
Reported by: jkim
Tested by: jkim
Sponsored by: Chelsio Communications
Add missing break to prevent falling through to the default case statement
and returning EINVAL for all session configs.
Sponsored by: Ampere Computing
Submitted by: Klara, Inc.
Currently, this supports SHA1 and SHA2-{224,256,384,512} both as plain
hashes and in HMAC mode on both amd64 and i386. It uses the SHA
intrinsics when present similar to aesni(4), but uses SSE/AVX
instructions when they are not.
Note that some files from OpenSSL that normally wrap the assembly
routines have been adapted to export methods usable by 'struct
auth_xform' as is used by existing software crypto routines.
Reviewed by: gallatin, jkim, delphij, gnn
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D26821
This patch adds support for IPsec ESN (Extended Sequence Numbers) in
encrypt and authenticate mode (eg. AES-CBC and SHA256) and combined mode
(eg. AES-GCM).
For the encrypt and authenticate mode the ESN is stored in separate
crp_esn buffer because the high-order 32 bits of the sequence number are
appended after the Next Header (RFC 4303).
For the combined modes the high-order 32 bits of the sequence number
[e.g. RFC 4106, Chapter 5 AAD Construction] are part of crp_aad
(prepared by netipsec layer in case of ESN support enabled), therefore
non visible diff around combined modes.
Submitted by: Grzegorz Jaszczyk <jaz@semihalf.com>
Patryk Duda <pdk@semihalf.com>
Reviewed by: jhb
Differential revision: https://reviews.freebsd.org/D22365
Obtained from: Semihalf
Sponsored by: Stormshield
Weirdly, I needed to sprinkle more parens here to get gcc-as in 6.4
to correctly generate things.
Without them, I'd get an unknown variable reference to SKEIN_ASM_UNROLL1024.
This at least links now, but I haven't run any test cases against it.
It may be worthwhile doing it in case gcc-as demands we liberally sprinkle
more brackets around variables in .if statements.
Thanks to ed for the suggestion of just sprinkling more brackets to
see if that helped.
Reviewed by: emaste
