gcm_*_aesni() are used when the AVX512 implementation is not available.
Fix two bugs which manifest when handling operations spanning multiple
segments:
- Avoid underflow when the length of the input is smaller than the
residual.
- In gcm_decrypt_aesni(), ensure that we begin the operation at the
right offset into the input and output buffers.
Reviewed by: jhb
Fixes: 9b1d87286c ("ossl: Add a fallback AES-GCM implementation using AES-NI")
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D42838
(cherry picked from commit 47d767dab54895f3ba8abac6ab2295797394659e)
ossl(4)'s AES-GCM implementation keeps mutable state in the session
structure, together with the key schedule. This was done for
convenience, as both are initialized together. However, some OCF
consumers, particularly ZFS, assume that requests may be dispatched to
the same session in parallel. Without serialization, this results in
incorrect output.
Fix the problem by explicitly copying per-session state onto the stack
at the beginning of each operation.
PR: 275306
Reviewed by: jhb
Fixes: 9a3444d91c ("ossl: Add a VAES-based AES-GCM implementation for amd64")
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D42783
(cherry picked from commit 5c0dac0b7a012f326edab06ad85aee5ad68ff120)
Consumers may optionally provide a reference to a separate buffer
containing AAD, but ossl_aes_gcm() didn't handle this and would thus
compute an incorrect digest.
Fixes: 9a3444d91c ("ossl: Add a VAES-based AES-GCM implementation for amd64")
Reviewed by: jhb
MFC after: 3 days
Sponsored by: Klara, Inc.
Sponsored by: Stormshield
Differential Revision: https://reviews.freebsd.org/D42736
(cherry picked from commit 87826c87c63b2995cb69e9c1a624c7e20dd70fcd)
OpenSSL 3.0.12 addresses:
* Fix incorrect key and IV resizing issues when calling
EVP_EncryptInit_ex2(), EVP_DecryptInit_ex2() or EVP_CipherInit_ex2()
with OSSL_PARAM parameters that alter the key or IV length
([CVE-2023-5363]).
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
(cherry picked from commit ad991e4c142ebabad7aef488ad97b189ecabb270)
(cherry picked from commit 575878a533823aa3e5bab715928d9cdffbc4dcbc)
There is a bug in the OpenSSL script which generates this file; the bug
is in the process of being fixed upstream.
Specifically, when generating the output, bsaes-armv7.pl strips some
labels that are used when the output asm is compiled with __KERNEL__
defined, resulting in a build error. As a step towards adding armv7
support to ossl(4), manually patch the generated asm. The upstream fix
will be imported later.
Reviewed by: andrew, jhb, emaste
MFC after: 1 week
Sponsored by: Klara, Inc.
Sponsored by: Stormshield
Differential Revision: https://reviews.freebsd.org/D41303
Chacha20+Poly1305 doesn't use an ossl_cipher instance the way AES-GCM
does, so ossl_lookup_cipher() failed causing ossl_newsession() to
always fail for Chacha20+Poly1305 sessions.
Reported by: gallatin (ktls_test fails with ossl.ko loaded)
Fixes: 9a3444d91c ossl: Add a VAES-based AES-GCM implementation for amd64
Tested by: gallatin
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D40580
This is required on i386. The patch has no functional change, since
AES-GCM isn't implemented for that platform.
Fixes: 9a3444d91c ("ossl: Add a VAES-based AES-GCM implementation for amd64")
Reported by: Jenkins
This lets one use ossl(4) for AES-GCM operations on contemporary amd64
platforms. A kernel benchmark indicates that this gives roughly
equivalent throughput to aesni(4) for various buffer sizes.
Bulk processing is done in aesni-gcm-x86_64.S, the rest is handled in a
C wrapper ported from OpenSSL's gcm128.c.
Sponsored by: Stormshield
Sponsored by: Klara, Inc.
Reviewed by: jhb
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D39967
aes-gcm-avx512.S is generated from OpenSSL 3.1 and implements AES-GCM.
ossl_x86.c detects whether the CPU implements the required AVX512
instructions; if not, the ossl(4) module does not provide an AES-GCM
implementation. The VAES implementation increases throughput for all
buffer sizes in both directions, up to 2x for sufficiently large
buffers.
The "process" implementation is in two parts: a generic OCF layer in
ossl_aes.c that calls a set of MD functions to do the heavy lifting.
The intent there is to make it possible to add other implementations for
other platforms, e.g., to reduce the diff required for D37421.
A follow-up commit will add a fallback path to legacy AES-NI, so that
ossl(4) can be used in preference to aesni(4) on all amd64 platforms.
In the long term we would like to replace aesni(4) and armv8crypto(4)
with ossl(4).
Note, currently this implementation will not be selected by default
since aesni(4) and ossl(4) return the same probe priority for crypto
sessions, and the opencrypto framework selects the first registered
implementation to break a tie. Since aesni(4) is compiled into the
kernel, aesni(4) wins. A separate change may modify ossl(4) to have
priority.
Sponsored by: Stormshield
Sponsored by: Klara, Inc.
Reviewed by: jhb
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D39783
This is needed to let OpenSSL 3.1 routines detect VAES and VPCLMULQDQ
extensions. The intent is to import ASM routines which implement
AES-GCM using VEX-prefixed AES-NI instructions.
No functional change intended.
Sponsored by: Stormshield
Sponsored by: Klara, Inc.
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D39782
The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.
Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix
All the other functions used pointers for from/to instead of
fixed-size array parameters. More importantly, this function can
accept pointers to buffers of multiple blocks, not just a single
block.
Reported by: GCC -Warray-parameter
Reviewed by: imp, emaste
Differential Revision: https://reviews.freebsd.org/D37547
OpenZFS changed the define _OPENSOLARIS_SYS_TYPES_H_ to
_SPL_SYS_TYPES_H_ to guard the sys/types.h compatibility file
inclusion. Follow the change here. The only place in the tree
_OPENSOLARIS_SYS_TYPES_H_ is mentioned is in the /*
_OPENSOLARIS_SYS_TYPES_H_ */ at the end of sys/types.h. That needs to be
changed upstream in OpenZFS since we don't like changing things in
FreeBSD's tree.
Sponsored by: Netflix
Reviewed by: tsoome, delphij
Differential Revision: https://reviews.freebsd.org/D35891
All supported compilers (modern versions of GCC and clang) support
this.
Many places didn't have an #else so would just silently do the wrong
thing. Ancient versions of icc (the original motivation for this) are
no longer a compiler FreeBSD supports.
PR: 263102 (exp-run)
Reviewed by: brooks, imp
Differential Revision: https://reviews.freebsd.org/D34797
Currently armv8crypto copies the scheme used in aesni(9), where payload
data and output buffers are allocated on the fly if the crypto buffer is
not virtually contiguous. This scheme is simple but incurs a lot of
overhead: for an encryption request with a separate output buffer we
have to
- allocate a temporary buffer to hold the payload
- copy input data into the buffer
- copy the encrypted payload to the output buffer
- zero the temporary buffer before freeing it
We have a handy crypto buffer cursor abstraction now, so reimplement the
armv8crypto routines using that instead of temporary buffers. This
introduces some extra complexity, but gallatin@ reports a 10% throughput
improvement with a KTLS workload without additional CPU usage. The
driver still allocates an AAD buffer for AES-GCM if necessary.
Reviewed by: jhb
Tested by: gallatin
Sponsored by: Ampere Computing LLC
Submitted by: Klara Inc.
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D28950
This is in preparation for using buffer cursors. No functional change
intended.
Reviewed by: jhb
Sponsored by: Ampere Computing LLC
Submitted by: Klara Inc.
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D28948
This adds a wrapper around libsodium's curve25519 support matching
Linux's curve25519 API. The intended use case for this is WireGuard.
Note that this is not integrated with OCF as it is not related to
symmetric operations on data.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33935
This is a synchronous software API which wraps the existing software
implementation shared with OCF. Note that this will not currently
use optimized backends (such as ossl(4)) but may be appropriate for
operations on small buffers.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33524
These callbacks allow multiple contiguous blocks to be manipulated in
a single call. Note that any trailing partial block for a stream
cipher must still be passed to encrypt/decrypt_last.
While here, document the setkey and reinit hooks and reorder the hooks
in 'struct enc_xform' to better reflect the life cycle.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33529
This is the same as 4285655adb ("aesni: Avoid a potential
out-of-bounds load in AES_GCM_encrypt()") except for the decryption
path.
Reported by: Jenkins (KASAN job)
Reviewed by: cem
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33474
As with sha256 add support for accelerated sha512 support to libmd on
arm64. This depends on clang 13+ to build as this is the first release
with the needed intrinsics. Gcc should also support them, however from
a currently unknown release.
Reviewed by: cem
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33373
Only pre-allocate auth contexts when a session-wide key is provided or
for sessions without keys. For sessions with per-operation keys,
always initialize the on-stack context directly rather than
initializing the session context in swcr_authprepare (now removed) and
then copying that session context into the on-stack context.
This approach permits parallel auth operations without needing a
serializing lock. In addition, the previous code assumed that auth
sessions always provided an initial key unlike cipher sessions which
assume either an initial key or per-op keys.
While here, fix the Blake2 auth transforms to function like other auth
transforms where Setkey is invoked after Init rather than before.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33316
This centralizes the check for valid nonce lengths for AES-GCM.
While here, remove some duplicate checks for valid AES-GCM tag lengths
from ccp(4) and ccr(4).
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33194
GCC 9 doesn't define a SHA256_Transform symbol when the stub just wraps
SHA256_Transform_c resulting in an undefined symbol for
_libmd_SHA256_Transform in libmd.so.
Discussed with: andrew, jrtc27
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D31945
Now that the AES-CBC is supported we can handle ETA requests.
Sponsored by: Stormshield
Obtained from: Semihalf
Reviewed by: jhb
Differential revision: https://reviews.freebsd.org/D32100
