mirror of
https://github.com/OISF/suricata.git
synced 2026-02-03 20:41:46 -05:00
9c324b796e
Ticket: #2696 There are a lot of changes here, which are described below. In general these changes are renaming constants to conform to the libhtp-rs versions (which are generated by cbindgen); making all htp types opaque and changing struct->member references to htp_struct_member() function calls; and a handful of changes to offload functionality onto libhtp-rs from suricata, such as URI normalization and transaction cleanup. Functions introduced to handle opaque htp_tx_t: - tx->parsed_uri => htp_tx_parsed_uri(tx) - tx->parsed_uri->path => htp_uri_path(htp_tx_parsed_uri(tx) - tx->parsed_uri->hostname => htp_uri_hostname(htp_tx_parsed_uri(tx)) - htp_tx_get_user_data() => htp_tx_user_data(tx) - htp_tx_is_http_2_upgrade(tx) convenience function introduced to detect response status 101 and “Upgrade: h2c" header. Functions introduced to handle opaque htp_tx_data_t: - d->len => htp_tx_data_len() - d->data => htp_tx_data_data() - htp_tx_data_tx(data) function to get the htp_tx_t from the htp_tx_data_t - htp_tx_data_is_empty(data) convenience function introduced to test if the data is empty. Other changes: Build libhtp-rs as a crate inside rust. Update autoconf to no longer use libhtp as an external dependency. Remove HAVE_HTP feature defines since they are no longer needed. Make function arguments and return values const where possible htp_tx_destroy(tx) will now free an incomplete transaction htp_time_t replaced with standard struct timeval Callbacks from libhtp now provide the htp_connp_t and the htp_tx_data_t as separate arguments. This means the connection parser is no longer fetched from the transaction inside callbacks. SCHTPGenerateNormalizedUri() functionality moved inside libhtp-rs, which now provides normalized URI values. The normalized URI is available with accessor function: htp_tx_normalized_uri() Configuration settings added to control the behaviour of the URI normalization: - htp_config_set_normalized_uri_include_all() - htp_config_set_plusspace_decode() - htp_config_set_convert_lowercase() - htp_config_set_double_decode_normalized_query() - htp_config_set_double_decode_normalized_path() - htp_config_set_backslash_convert_slashes() - htp_config_set_bestfit_replacement_byte() - htp_config_set_convert_lowercase() - htp_config_set_nul_encoded_terminates() - htp_config_set_nul_raw_terminates() - htp_config_set_path_separators_compress() - htp_config_set_path_separators_decode() - htp_config_set_u_encoding_decode() - htp_config_set_url_encoding_invalid_handling() - htp_config_set_utf8_convert_bestfit() - htp_config_set_normalized_uri_include_all() - htp_config_set_plusspace_decode() Constants related to configuring uri normalization: - HTP_URL_DECODE_PRESERVE_PERCENT => HTP_URL_ENCODING_HANDLING_PRESERVE_PERCENT - HTP_URL_DECODE_REMOVE_PERCENT => HTP_URL_ENCODING_HANDLING_REMOVE_PERCENT - HTP_URL_DECODE_PROCESS_INVALID => HTP_URL_ENCODING_HANDLING_PROCESS_INVALID htp_config_set_field_limits(soft_limit, hard_limit) changed to htp_config_set_field_limit(limit) because libhtp didn't implement soft limits. libhtp logging API updated to provide HTP_LOG_CODE constants along with the message. This eliminates the need to perform string matching on message text to map log messages to HTTP_DECODER_EVENT values, and the HTP_LOG_CODE values can be used directly. In support of this, HTP_DECODER_EVENT values are mapped to their corresponding HTP_LOG_CODE values. New log events to describe additional anomalies: HTP_LOG_CODE_REQUEST_TOO_MANY_LZMA_LAYERS HTP_LOG_CODE_RESPONSE_TOO_MANY_LZMA_LAYERS HTP_LOG_CODE_PROTOCOL_CONTAINS_EXTRA_DATA HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_START HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_END HTP_LOG_CODE_SWITCHING_PROTO_WITH_CONTENT_LENGTH HTP_LOG_CODE_DEFORMED_EOL HTP_LOG_CODE_PARSER_STATE_ERROR HTP_LOG_CODE_MISSING_OUTBOUND_TRANSACTION_DATA HTP_LOG_CODE_MISSING_INBOUND_TRANSACTION_DATA HTP_LOG_CODE_ZERO_LENGTH_DATA_CHUNKS HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_NO_PROTOCOL HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_INVALID_PROTOCOL HTP_LOG_CODE_REQUEST_LINE_NO_PROTOCOL HTP_LOG_CODE_RESPONSE_LINE_INVALID_PROTOCOL HTP_LOG_CODE_RESPONSE_LINE_INVALID_RESPONSE_STATUS HTP_LOG_CODE_RESPONSE_BODY_INTERNAL_ERROR HTP_LOG_CODE_REQUEST_BODY_DATA_CALLBACK_ERROR HTP_LOG_CODE_RESPONSE_INVALID_EMPTY_NAME HTP_LOG_CODE_REQUEST_INVALID_EMPTY_NAME HTP_LOG_CODE_RESPONSE_INVALID_LWS_AFTER_NAME HTP_LOG_CODE_RESPONSE_HEADER_NAME_NOT_TOKEN HTP_LOG_CODE_REQUEST_INVALID_LWS_AFTER_NAME HTP_LOG_CODE_LZMA_DECOMPRESSION_DISABLED HTP_LOG_CODE_CONNECTION_ALREADY_OPEN HTP_LOG_CODE_COMPRESSION_BOMB_DOUBLE_LZMA HTP_LOG_CODE_INVALID_CONTENT_ENCODING HTP_LOG_CODE_INVALID_GAP HTP_LOG_CODE_ERROR The new htp_log API supports consuming log messages more easily than walking a list and tracking the current offset. Internally, libhtp-rs now provides log messages as a queue of htp_log_t, which means the application can simply call htp_conn_next_log() to fetch the next log message until the queue is empty. Once the application is done with a log message, they can call htp_log_free() to dispose of it. Functions supporting htp_log_t: htp_conn_next_log(conn) - Get the next log message htp_log_message(log) - To get the text of the message htp_log_code(log) - To get the HTP_LOG_CODE value htp_log_free(log) - To free the htp_log_t
105 lines
2.7 KiB
Bash
Executable file
105 lines
2.7 KiB
Bash
Executable file
#! /usr/bin/env bash
|
|
#
|
|
# This script will bundle suricata-update for you.
|
|
#
|
|
# To use, run from the top Suricata source directory:
|
|
#
|
|
# ./scripts/bundle.sh [suricata-update]
|
|
#
|
|
# If no arguments are provided, suricata-update will
|
|
# be bundled.
|
|
#
|
|
# Environment variables:
|
|
#
|
|
# SU_REPO: Overrides the Suricata-Update git repo
|
|
# SU_BRANCH: Override the Suricata-Update branch to a branch, tag or
|
|
# {pull,merge}-request.
|
|
#
|
|
# DESTDIR: Checkout to another directory instead of the current
|
|
# directory.
|
|
#
|
|
# To specify a pull or merge request in a branch name in the format of
|
|
# pr/NNN or mr/NNN.
|
|
|
|
set -e
|
|
|
|
DESTDIR=${DESTDIR:-.}
|
|
|
|
what="$1"
|
|
|
|
# Transforms a branch name in the form of "pr/<NUMBER>" or
|
|
# "mr/<NUMBER>" into a proper ref for GitHub or GitLab.
|
|
|
|
# Transform a branch name to a ref.
|
|
#
|
|
# For GitHub the following formats are allowed:
|
|
# - pr/123
|
|
# - pull/123
|
|
# - https://github.com/OISF/suricata-update/pull/123
|
|
# - OISF/suricata-update#123
|
|
#
|
|
# For GibLab only the format "mr/123" is supported.
|
|
transform_branch() {
|
|
pr=$(echo "${1}" | sed -n \
|
|
-e 's/^https:\/\/github.com\/OISF\/.*\/pull\/\([0-9]*\)$/\1/p' \
|
|
-e 's/^OISF\/.*#\([0-9]*\)$/\1/p' \
|
|
-e 's/^pull\/\([0-9]*\)$/\1/p' \
|
|
-e 's/^pr\/\([0-9]*\)$/\1/p')
|
|
if [ "${pr}" ]; then
|
|
echo "refs/pull/${pr}/head"
|
|
return
|
|
fi
|
|
|
|
mr=$(echo "${1}" | sed -n 's/^mr\/\([[:digit:]]\+\)$/\1/p')
|
|
if [ "${mr}" ]; then
|
|
echo "refs/merge-requests/${mr}/head"
|
|
return
|
|
fi
|
|
|
|
echo "${1}"
|
|
}
|
|
|
|
fetch() {
|
|
repo="$1"
|
|
dest="$2"
|
|
branch="$3"
|
|
|
|
git clone --depth 1 "${repo}" "${dest}"
|
|
pushd "${dest}"
|
|
git fetch origin "${branch}"
|
|
git -c advice.detachedHead=false checkout FETCH_HEAD
|
|
popd
|
|
}
|
|
|
|
while IFS= read -r requirement; do
|
|
set -- $requirement
|
|
|
|
# If a requirement was specified on the command line, skip all other
|
|
# requirements.
|
|
if [ "${what}" != "" ]; then
|
|
if [ "${what}" != "$1" ]; then
|
|
continue
|
|
fi
|
|
fi
|
|
case "$1" in
|
|
suricata-update)
|
|
SU_REPO=${SU_REPO:-$2}
|
|
SU_BRANCH=$(transform_branch ${SU_BRANCH:-$3})
|
|
echo "===> Bundling ${SU_REPO} (${SU_BRANCH})"
|
|
rm -rf ${DESTDIR}/suricata-update.tmp
|
|
fetch "${SU_REPO}" "${DESTDIR}/suricata-update.tmp" "${SU_BRANCH}"
|
|
rm -rf ${DESTDIR}/suricata-update.tmp/.git
|
|
cp -a ${DESTDIR}/suricata-update.tmp/. ${DESTDIR}/suricata-update
|
|
rm -rf ${DESTDIR}/suricata-update.tmp
|
|
;;
|
|
\#*)
|
|
# Ignore comment.
|
|
;;
|
|
"")
|
|
# Ignore blank line.
|
|
;;
|
|
*)
|
|
echo "error: unknown requirement: $1"
|
|
;;
|
|
esac
|
|
done < ./requirements.txt
|