*wrong branch** Revert "prepare for 19.1, add etpro_telemetry"

This reverts commit e0de6940f4.
This commit is contained in:
Ad Schellevis 2019-01-25 19:28:25 +01:00
parent e0de6940f4
commit 76ed7724a9
16 changed files with 0 additions and 905 deletions

View file

@ -1,8 +0,0 @@
PLUGIN_NAME= etpro_telemetry
PLUGIN_VERSION= 1.0
#PLUGIN_REVISION= 1
PLUGIN_COMMENT= ETPRO telemetry edition
PLUGIN_MAINTAINER= ad@opnsense.org
PLUGIN_WWW= https://docs.opnsense.org/manual/etpro_telemetry.html
.include "../../Mk/plugins.mk"

View file

@ -1,20 +0,0 @@
Todays cybersecurity engineers need timely and accurate data about eminent threats and how they spread around the globe.
With this data cybersecurity researchers and analysts can improve the detection of malicious network traffic.
The times when we could rely on just firewall rules for our protection are long gone.
Additional layers of security are desperately needed to guard against these attacks.
With growing risks the need to fortify our security is growing for both big enterprises as for SMEs alike, but often
out of reach for the latter.
An important extra security addition is an Intrusion Detection and Prevention System (IDS/IPS).
The IDS/IPS available in OPNsense is based on Suricata.
This open source IDS/IPS engine has proven its value in OPNsense, especially in combination with the free Proofpoint ETOpen ruleset.
The need for valuable threat detection data and the increasing importance of additional network security
has brought Proofpoint and OPNsense together.
Our joined efforts resulted in the ETPro Telemetry edition.
The ETPro Telemetry edition embraces our vision that sharing knowledge leads to better products.
When you allow your OPNsense system to share anonymized information about detected threats - the alerts -
you are able to use the ETPro ruleset free of charge.

View file

@ -1,10 +0,0 @@
# DO NOT EDIT THIS FILE -- OPNsense auto-generated file
#
# User-defined crontab files can be loaded via /etc/cron.d
# or /usr/local/etc/cron.d and follow the same format as
# /etc/crontab, see the crontab(5) manual page.
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
#minute hour mday month wday who command
0,30 * * * * root /usr/local/opnsense/scripts/etpro_telemetry/send_heartbeat.py
* * * * * root /usr/local/opnsense/scripts/etpro_telemetry/send_telemetry.py

View file

@ -1,53 +0,0 @@
<?php
/*
* Copyright (C) 2018-2019 Deciso B.V.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* 1. Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
*
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
* INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
* AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
* AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
* OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
namespace OPNsense\Diagnostics\Api;
use \OPNsense\Base\ApiControllerBase;
use \OPNsense\Core\Backend;
/**
* Class ProofpointEtController
* @package OPNsense\Diagnostics\Api
*/
class ProofpointEtController extends ApiControllerBase
{
/**
* retrieve system activity (top)
* @return array
*/
public function statusAction()
{
$this->sessionClose();
$backend = new Backend();
$response = $backend->configdRun('proofpoint et status');
$activity = json_decode($response, true);
return $activity;
}
}

View file

@ -1,60 +0,0 @@
#!/usr/local/bin/python2.7
"""
Copyright (c) 2018-2019 Ad Schellevis <ad@opnsense.org>
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
"""
import argparse
import urllib3
import datetime
import ujson
import telemetry.log
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
parser = argparse.ArgumentParser()
parser.add_argument('-l', '--log', help='log directory containing eve.json files', default="/var/log/suricata/")
parser.add_argument('-t', '--time', help='max seconds to read from now()', type=int, default=3600)
parser.add_argument('-p', '--parsed', help='show data as shipped using send_telemetry',
default=False, action="store_true")
parser.add_argument('-L', '--limit', help='limit number of rows', type=int, default=-1)
args = parser.parse_args()
last_update = datetime.datetime.now() - datetime.timedelta(seconds=float(args.time))
event_collector = telemetry.EventCollector()
row_count = 0
for record in telemetry.log.reader(args.log, last_update=last_update):
if args.parsed:
event_collector.push(record)
else:
print (ujson.dumps(record))
row_count += 1
if args.limit != -1 and row_count >= args.limit:
break
if args.parsed:
for push_data in event_collector:
print (push_data.strip())

View file

@ -1,71 +0,0 @@
#!/usr/local/bin/python2.7
"""
Copyright (c) 2018-2019 Ad Schellevis <ad@opnsense.org>
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
"""
import sys
import argparse
import requests
import syslog
import urllib3
import telemetry
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
parser = argparse.ArgumentParser()
parser.add_argument('-e', '--endpoint',
help='Endpoint url to reach',
default="%s/api/v1/sensor" % telemetry.BASE_URL)
parser.add_argument('-i', '--insecure',
help='Insecure, skip certificate validation',
action="store_true",
default=False)
parser.add_argument('-c', '--config',
help='rule downloader configuration',
default="/usr/local/etc/suricata/rule-updater.config"
)
args = parser.parse_args()
exit_code = -1
cnf = telemetry.get_config(args.config)
if cnf.token is not None:
params = {'timeout': 5, 'headers': {'Authorization': 'Bearer %s' % cnf.token}}
if args.insecure:
params['verify'] = False
try:
r = requests.head(args.endpoint, **params)
if r.status_code == 200:
# expected result, set exit code
exit_code = 0
else:
syslog.syslog(syslog.LOG_ERR, 'unexpected result from %s (http_code %s)' % (args.endpoint, r.status_code))
except requests.exceptions.ConnectionError:
syslog.syslog(syslog.LOG_ERR, 'connection error sending heardbeat to %s' % args.endpoint)
else:
syslog.syslog(syslog.LOG_ERR, 'telemetry token missing in %s' % args.config)
# exit
sys.exit(exit_code)

View file

@ -1,119 +0,0 @@
#!/usr/local/bin/python2.7
"""
Copyright (c) 2018-2019 Ad Schellevis <ad@opnsense.org>
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
"""
import sys
import os
import argparse
import requests
import time
import syslog
import urllib3
import ujson
import telemetry.log
import telemetry.state
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
parser = argparse.ArgumentParser()
parser.add_argument('-e', '--endpoint', help='Endpoint url to reach',
default="%s/api/v1/event" % telemetry.BASE_URL)
parser.add_argument('-i', '--insecure', help='Insecure, skip certificate validation',
action="store_true", default=False)
parser.add_argument('-c', '--config', help='rule downloader configuration',
default="/usr/local/etc/suricata/rule-updater.config")
parser.add_argument('-l', '--log', help='log directory containing eve.json files',
default="/var/log/suricata/")
parser.add_argument('-s', '--state', help='persistent state (and lock) filename',
default="/usr/local/var/run/et_telemetry.state")
parser.add_argument('-d', '--days', help='Maximum number of days to look back on initial run', type=int, default=2)
args = parser.parse_args()
exit_code = -1
send_start_time = time.time()
telemetry_state = telemetry.state.Telemetry(filename=args.state, init_last_days=args.days)
if not telemetry_state.is_running():
cnf = telemetry.get_config(args.config)
if cnf.token is not None:
if os.path.isdir(args.log):
last_update = telemetry_state.get_last_update()
event_collector = telemetry.EventCollector()
row_count = 0
max_timestamp = None
for record in telemetry.log.reader(args.log, last_update):
if max_timestamp is None or record['__timestamp__'] > max_timestamp:
max_timestamp = record['__timestamp__']
event_collector.push(record)
row_count += 1
# data collected, log and push
if row_count > 0 and max_timestamp is not None:
syslog.syslog(
syslog.LOG_NOTICE,
'telemetry data collected %d records in %.2f seconds @%s' % (
row_count, time.time() - send_start_time, max_timestamp
)
)
for push_data in event_collector:
params = {
'timeout': 5,
'headers': {'Authorization': 'Bearer %s' % cnf.token},
'data': push_data.strip()
}
if args.insecure:
params['verify'] = False
r = requests.post(args.endpoint, **params)
if r.status_code != 201:
syslog.syslog(
syslog.LOG_ERR,
'unexpected result from %s (http_code %s)' % (args.endpoint, r.status_code)
)
exit_code = -1
break
else:
try:
ujson.loads(r.text)
exit_code = 0
except ValueError:
syslog.syslog(syslog.LOG_ERR, 'telemetry unexpected response %s' % r.text[:256])
exit_code = -1
break
if exit_code == 0:
# update timestamp, last record processed
telemetry_state.set_last_update(max_timestamp)
else:
# no data
exit_code = 0
else:
syslog.syslog(syslog.LOG_ERR, 'telemetry token missing in %s' % args.config)
else:
syslog.syslog(syslog.LOG_ERR, 'directory %s missing' % args.log)
sys.exit(exit_code)

View file

@ -1,61 +0,0 @@
#!/usr/local/bin/python2.7
"""
Copyright (c) 2018-2019 Ad Schellevis <ad@opnsense.org>
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
"""
import argparse
import requests
import urllib3
import ujson
import telemetry
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
parser = argparse.ArgumentParser()
parser.add_argument('-e', '--endpoint', help='Endpoint url to reach',
default="%s/api/v1/sensorinfo" % telemetry.BASE_URL)
parser.add_argument('-i', '--insecure', help='Insecure, skip certificate validation',
action="store_true", default=False)
parser.add_argument('-c', '--config', help='rule downloader configuration',
default="/usr/local/etc/suricata/rule-updater.config")
args = parser.parse_args()
cnf = telemetry.get_config(args.config)
if cnf.token is not None:
try:
req = requests.get(args.endpoint, headers={'Authorization': 'Bearer %s' % cnf.token})
if req.status_code == 200:
response = ujson.loads(req.text)
response['status'] = 'ok'
else:
response = {'status': 'failed', 'response': req.text}
except requests.exceptions.SSLError as e:
response = {'status': 'failed', 'response': '%s' % e}
except ValueError:
response = {'status': 'failed', 'response': req.text}
else:
response = {'status': 'unconfigured'}
print (ujson.dumps(response))

View file

@ -1,140 +0,0 @@
"""
Copyright (c) 2018-2019 Ad Schellevis <ad@opnsense.org>
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
"""
import os
import subprocess
import tempfile
import collections
try:
from configparser import ConfigParser
except ImportError:
from ConfigParser import ConfigParser
import netaddr
import ujson
BASE_URL = 'https://opnsense.emergingthreats.net'
def get_config(rule_update_config):
"""
retrieve device token, since we align our telemetry data to the existing rule download feature in OPNsense
it should be safe to assume rule-updater.config contains the token that is used.
:param rule_update_config: path to OPNsense rule update configuration
:return: token id or None if not found
"""
response = collections.namedtuple('sensor', 'token')
if os.path.exists(rule_update_config):
cnf = ConfigParser()
cnf.read(rule_update_config)
if cnf.has_section('__properties__'):
if cnf.has_option('__properties__', 'et_telemetry.token'):
response.token = cnf.get('__properties__', 'et_telemetry.token')
return response
class EventCollector(object):
""" Event collector, responsible for extracting and anonymising from an eve.json stream
"""
def __init__(self):
self._tmp_handle = tempfile.NamedTemporaryFile()
self._local_networks = list()
self._get_local_networks()
def _get_local_networks(self):
""" collect local attached networks for anonymization purposes
:return: None
"""
with tempfile.NamedTemporaryFile() as output_stream:
subprocess.call(['ifconfig', '-a'], stdout=output_stream, stderr=open(os.devnull, 'wb'))
output_stream.seek(0)
for line in output_stream:
if line.startswith(b'\tinet'):
parts = line.split()
if len(parts) > 3:
if parts[0] == 'inet6' and parts[2] == 'prefixlen':
# IPv6 addresses
self._local_networks.append(
netaddr.IPNetwork("%s/%s" % (parts[1].split('%')[0], parts[3]))
)
elif parts[0] == 'inet' and len(parts) > 3 and parts[2] == 'netmask':
# IPv4 addresses
mask = int(parts[3], 16)
self._local_networks.append(
netaddr.IPNetwork("%s/%s" % (netaddr.IPAddress(parts[1]), netaddr.IPAddress(mask)))
)
def is_local_address(self, address):
""" check if provided address is local for this device
:param address: address (string)
:return: boolean
"""
addr_to_check = netaddr.IPAddress(address)
for local_network in self._local_networks:
if addr_to_check in local_network:
return True
return False
def push(self, record):
""" cleanup and write record
:param record: parsed eve log record
:return: None
"""
to_push = dict()
for address in ['src_ip', 'dest_ip']:
if address in record:
if self.is_local_address(record[address]):
if record[address].find(':') > -1:
# replace local IPv6 address
to_push[address] = 'xxxx:xxxx:%s' % ':'.join(record[address].split(':')[-2:])
else:
to_push[address] = 'xxx.xxx.xxx.%s' % record[address].split('.')[-1]
else:
# non local address
to_push[address] = record[address]
# unfiltered output fields
for attr in ["timestamp", "flow_id", "in_iface", "event_type",
"vlan", "src_port", "dest_port", "proto", "alert"]:
if attr in record:
to_push[attr] = record[attr]
self._tmp_handle.write(("%s\n" % ujson.dumps(to_push)).encode())
def get(self):
""" fetch all data from temp
:return:
"""
self._tmp_handle.seek(0)
return self._tmp_handle.read()
def __iter__(self):
""" Iterate parsed events
:return:
"""
self._tmp_handle.seek(0)
for line in self._tmp_handle:
yield line

View file

@ -1,105 +0,0 @@
"""
Copyright (c) 2018-2019 Ad Schellevis <ad@opnsense.org>
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
"""
import os
import re
import glob
import datetime
import ujson
def reverse_log_reader(filename):
""" read log file in reverse order
:param filename: filename
:return: generator
"""
block_size = 81920
input_stream = open(filename, 'rU')
input_stream.seek(0, os.SEEK_END)
file_byte_start = input_stream.tell()
data = ''
while file_byte_start > 0:
if file_byte_start - block_size < 0:
block_size = file_byte_start
file_byte_start = 0
else:
file_byte_start -= block_size
input_stream.seek(file_byte_start)
data = input_stream.read(block_size) + data
bol = data.rfind('\n')
eol = len(data)
while bol > -1:
yield data[bol:eol]
eol = bol
bol = data.rfind('\n', 0, eol)
data = data[:eol] if bol == -1 else ''
if file_byte_start == 0 and bol == -1:
yield data
def parse_log_line(line):
""" parse eve log records, add __timestamp__ containing a timezoned datetime object
:param line: raw json string
:return: dict or None if unable to parse
"""
try:
record = ujson.loads(line.strip())
except ValueError:
record = dict()
if 'timestamp' in record:
# convert timestamp
try:
tmp = list(map(lambda x: int(x), re.split('T|-|\:|\.|\+', record['timestamp'])))
tz = record['timestamp'][-5:]
ts = datetime.datetime(*tmp[:7])
ts -= datetime.timedelta(hours=int(tz[2:3]), minutes=int(tz[-2:])) * int(tz[0:1] + '1')
record['__timestamp__'] = ts
return record
except ValueError:
pass
return None
def reader(log_directory, last_update):
""" read and parse eve logs until timestamp is reached
:param log_directory: directory to search for eve.json*
:param last_update: datetime of last send event
:return: iterator
"""
for filename in sorted(glob.glob("%s/eve.json*" % log_directory)):
for line in reverse_log_reader(filename=filename):
record = parse_log_line(line)
if record:
if record['__timestamp__'] > last_update:
yield record
else:
return

View file

@ -1,71 +0,0 @@
"""
Copyright (c) 2018-2019 Ad Schellevis <ad@opnsense.org>
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
"""
import fcntl
import datetime
class Telemetry(object):
def __init__(self, filename, init_last_days=2):
self._filename = filename
self._init_last_days = init_last_days
try:
self._file_handle = open(self._filename, 'a+')
fcntl.flock(self._file_handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
except IOError:
# already running
self._file_handle = None
def is_running(self):
return self._file_handle is None
def get_last_update(self):
""" return last used timestamp
:return: datetime
"""
self._file_handle.seek(0)
try:
result = datetime.datetime.fromtimestamp(float(self._file_handle.readline()))
except ValueError:
result = datetime.datetime.now() - datetime.timedelta(days=self._init_last_days)
return result
def set_last_update(self, stamp):
""" set last timestamp
:param stamp: datetime object
:return:
"""
self._file_handle.seek(0)
self._file_handle.truncate()
self._file_handle.write("%s\n" % stamp.strftime('%s.%f'))
def __del__(self):
""" close file handle on destruct
:return:
"""
if self._file_handle:
self._file_handle.close()

View file

@ -1,58 +0,0 @@
<?xml version="1.0"?>
<ruleset documentation_url="http://docs.opnsense.org/">
<location url="https://opnsense.emergingthreats.net/api/v1/ruleset" prefix="ET telemtry"/>
<headers>
<Authorization>Bearer %%et_telemetry.token%%</Authorization>
</headers>
<version url="https://opnsense.emergingthreats.net/api/v1/ruleset/version"/>
<files>
<file description="botcc.portgrouped" url="inline::rules/botcc.portgrouped.rules">botcc.portgrouped.rules</file>
<file description="botcc" url="inline::rules/botcc.rules">botcc.rules</file>
<file description="ciarmy" url="inline::rules/ciarmy.rules">ciarmy.rules</file>
<file description="compromised" url="inline::rules/compromised.rules">compromised.rules</file>
<file description="drop" url="inline::rules/drop.rules">drop.rules</file>
<file description="dshield" url="inline::rules/dshield.rules">dshield.rules</file>
<file description="emerging-activex" url="inline::rules/emerging-activex.rules">emerging-activex.rules</file>
<file description="emerging-attack_response" url="inline::rules/emerging-attack_response.rules">emerging-attack_response.rules</file>
<file description="emerging-chat" url="inline::rules/emerging-chat.rules">emerging-chat.rules</file>
<file description="emerging-current_events" url="inline::rules/emerging-current_events.rules">emerging-current_events.rules</file>
<file description="emerging-deleted" url="inline::rules/emerging-deleted.rules">emerging-deleted.rules</file>
<file description="emerging-dns" url="inline::rules/emerging-dns.rules">emerging-dns.rules</file>
<file description="emerging-dos" url="inline::rules/emerging-dos.rules">emerging-dos.rules</file>
<file description="emerging-exploit" url="inline::rules/emerging-exploit.rules">emerging-exploit.rules</file>
<file description="emerging-ftp" url="inline::rules/emerging-ftp.rules">emerging-ftp.rules</file>
<file description="emerging-games" url="inline::rules/emerging-games.rules">emerging-games.rules</file>
<file description="emerging-icmp" url="inline::rules/emerging-icmp.rules">emerging-icmp.rules</file>
<file description="emerging-icmp_info" url="inline::rules/emerging-icmp_info.rules">emerging-icmp_info.rules</file>
<file description="emerging-imap" url="inline::rules/emerging-imap.rules">emerging-imap.rules</file>
<file description="emerging-inappropriate" url="inline::rules/emerging-inappropriate.rules">emerging-inappropriate.rules</file>
<file description="emerging-info" url="inline::rules/emerging-info.rules">emerging-info.rules</file>
<file description="emerging-malware" url="inline::rules/emerging-malware.rules">emerging-malware.rules</file>
<file description="emerging-misc" url="inline::rules/emerging-misc.rules">emerging-misc.rules</file>
<file description="emerging-mobile_malware" url="inline::rules/emerging-mobile_malware.rules">emerging-mobile_malware.rules</file>
<file description="emerging-netbios" url="inline::rules/emerging-netbios.rules">emerging-netbios.rules</file>
<file description="emerging-p2p" url="inline::rules/emerging-p2p.rules">emerging-p2p.rules</file>
<file description="emerging-policy" url="inline::rules/emerging-policy.rules">emerging-policy.rules</file>
<file description="emerging-pop3" url="inline::rules/emerging-pop3.rules">emerging-pop3.rules</file>
<file description="emerging-rpc" url="inline::rules/emerging-rpc.rules">emerging-rpc.rules</file>
<file description="emerging-scada" url="inline::rules/emerging-scada.rules">emerging-scada.rules</file>
<file description="emerging-scan" url="inline::rules/emerging-scan.rules">emerging-scan.rules</file>
<file description="emerging-shellcode" url="inline::rules/emerging-shellcode.rules">emerging-shellcode.rules</file>
<file description="emerging-smtp" url="inline::rules/emerging-smtp.rules">emerging-smtp.rules</file>
<file description="emerging-snmp" url="inline::rules/emerging-snmp.rules">emerging-snmp.rules</file>
<file description="emerging-sql" url="inline::rules/emerging-sql.rules">emerging-sql.rules</file>
<file description="emerging-telnet" url="inline::rules/emerging-telnet.rules">emerging-telnet.rules</file>
<file description="emerging-tftp" url="inline::rules/emerging-tftp.rules">emerging-tftp.rules</file>
<file description="emerging-trojan" url="inline::rules/emerging-trojan.rules">emerging-trojan.rules</file>
<file description="emerging-user_agents" url="inline::rules/emerging-user_agents.rules">emerging-user_agents.rules</file>
<file description="emerging-voip" url="inline::rules/emerging-voip.rules">emerging-voip.rules</file>
<file description="emerging-web_client" url="inline::rules/emerging-web_client.rules">emerging-web_client.rules</file>
<file description="emerging-web_server" url="inline::rules/emerging-web_server.rules">emerging-web_server.rules</file>
<file description="emerging-web_specific_apps" url="inline::rules/emerging-web_specific_apps.rules">emerging-web_specific_apps.rules</file>
<file description="emerging-worm" url="inline::rules/emerging-worm.rules">emerging-worm.rules</file>
<file description="tor" url="inline::rules/tor.rules">tor.rules</file>
</files>
<properties>
<property name="et_telemetry.token" default=""/>
</properties>
</ruleset>

View file

@ -1,5 +0,0 @@
[et.status]
command:/usr/local/opnsense/scripts/etpro_telemetry/sensor_info.py
parameters:
type:script_output
message:proofpoint ids telemetry status

View file

@ -1,26 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 519.18 96.99">
<defs>
<style>
.cls-1 {
fill: #231f20;
}
</style>
</defs>
<title>Asset 1</title>
<g id="Layer_2" data-name="Layer 2">
<g id="Layer_1-2" data-name="Layer 1">
<path class="cls-1" d="M499.39,75.36V64.28h-3.83c-4.16,0-5.14-.62-5.14-3.52V30.33h9V20h-9V5H475V20h-7.66V30.33H475V60c0,11.49,2.74,15.63,15.54,15.63,1.64,0,4.6-.21,8.87-.31"/>
<path class="cls-1" d="M410.91,74.94h15.54V42.34c0-7.25,4.71-11.59,11.93-11.59,6.67,0,9,3.52,9,11V74.94h15.54V37.47c0-12.63-6.35-18.94-19.15-18.94-7.34,0-13.24,2.69-17.62,8V20H410.91Z"/>
<rect class="cls-1" x="385.84" y="0.52" width="15.54" height="13.35"/>
<rect class="cls-1" x="385.84" y="19.98" width="15.54" height="54.96"/>
<path class="cls-1" d="M336.71,47.41c0-11,5-17,13.68-17s13.57,6,13.57,17S359,64.49,350.38,64.49s-13.68-6.11-13.68-17.08m-15.76,0c0,18.32,11.16,29.39,29.44,29.39s29.44-11.07,29.44-29.39S368.66,18,350.38,18s-29.44,11.07-29.44,29.4"/>
<path class="cls-1" d="M300.49,47.82c0,10.25-4.92,16.25-12.59,16.25-8.21,0-13-5.8-13-16.25,0-11.28,4.49-17.08,12.7-17.08s12.91,5.9,12.91,17.08M259.77,97h15.1V68.11c3.83,5.69,9.74,8.69,17.18,8.69,13.57,0,23.86-11.49,23.86-29.6,0-17.6-10-29.09-24.08-29.09-7.55,0-12.8,2.9-17.29,9.32V20H259.77Z"/>
<path class="cls-1" d="M228.22,74.94h15.21V30.33h9.63V20h-9.74V16.15c0-3.52,1.42-4.14,5.69-4.14h4.05V.31C250.33.1,247.93,0,245.63,0c-11.93,0-17.4,4.56-17.4,14.91V20h-8.32V30.33h8.32Z"/>
<path class="cls-1" d="M175.81,47.41c0-11,5-17,13.68-17s13.57,6,13.57,17-4.92,17.08-13.57,17.08-13.68-6.11-13.68-17.08m-15.76,0c0,18.32,11.16,29.39,29.44,29.39s29.44-11.07,29.44-29.39S207.76,18,189.48,18,160,29.09,160,47.41"/>
<path class="cls-1" d="M113.38,47.41c0-11,5-17,13.68-17s13.57,6,13.57,17-4.92,17.08-13.57,17.08-13.68-6.11-13.68-17.08m-15.76,0c0,18.32,11.16,29.39,29.44,29.39S156.5,65.73,156.5,47.41,145.34,18,127.06,18,97.62,29.09,97.62,47.41"/>
<path class="cls-1" d="M62.38,74.94H77.92V45.85c0-8.28,4.38-12.42,12.8-12.42h5.14V18.53a17.07,17.07,0,0,0-2.3-.1c-7.44,0-12.7,3.41-16.75,11V20H62.38Z"/>
<path class="cls-1" d="M40.71,47.82c0,10.25-4.92,16.25-12.59,16.25-8.21,0-13-5.8-13-16.25,0-11.28,4.49-17.08,12.7-17.08s12.91,5.9,12.91,17.08M0,97H15.1V68.11c3.83,5.69,9.74,8.69,17.18,8.69,13.57,0,23.86-11.49,23.86-29.6,0-17.6-10-29.09-24.08-29.09-7.55,0-12.8,2.9-17.29,9.32V20H0Z"/>
<path class="cls-1" d="M519.18,70.4a6.07,6.07,0,1,1-6-5.9A5.93,5.93,0,0,1,519.18,70.4Zm-10.62,0a4.57,4.57,0,0,0,4.61,4.72,4.52,4.52,0,0,0,4.5-4.68,4.56,4.56,0,1,0-9.11,0Zm3.64,3.1h-1.37V67.6a11.93,11.93,0,0,1,2.27-.18,3.68,3.68,0,0,1,2.05.43,1.64,1.64,0,0,1,.58,1.3,1.51,1.51,0,0,1-1.22,1.37v.07A1.74,1.74,0,0,1,515.58,72,4.52,4.52,0,0,0,516,73.5h-1.48a5,5,0,0,1-.47-1.44c-.11-.65-.47-.94-1.22-.94h-.65Zm0-3.35h.65c.76,0,1.37-.25,1.37-.86s-.4-.9-1.26-.9a3.26,3.26,0,0,0-.76.07Z"/>
</g>
</g>
</svg>

Before

Width:  |  Height:  |  Size: 2.9 KiB

View file

@ -1,2 +0,0 @@
<?php
$proofpoint_et_title = gettext('Telemetry status');

View file

@ -1,96 +0,0 @@
<?php
/*
Copyright (C) 2018-2019 Deciso B.V.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
*/
?>
<style>
.proofpoint_logo {
height: 30px;
vertical-align: middle;
}
.proofpoint_status_div {
padding: 2px;
border: 2px solid black;
white-space: nowrap;
display: inline;
}
</style>
<script src="/ui/js/moment-with-locales.min.js"></script>
<script>
$(window).load(function() {
ajaxGet("/api/diagnostics/proofpoint_et/status", {}, function(data){
$("#proofpoint_status").removeClass("fa-spin");
$("#proofpoint_status").removeClass("fa-spinner");
if (data.status == 'ok' && data.sensor_status == 'ACTIVE') {
$("#proofpoint_status").addClass("fa-check");
} else {
$("#proofpoint_status").addClass("fa-close");
}
if (data.status == 'ok') {
var props = {
sensor_status: "<?=html_safe(gettext('Status'));?>",
event_received: "<?=html_safe(gettext('Last event'));?>",
last_rule_download: "<?=html_safe(gettext('Last rule download'));?>",
last_heartbeat: "<?=html_safe(gettext('Last heartbeat'));?>"
};
$("#proofpoint_status_table > tbody").empty();
for (var idx in props) {
if (idx === 'sensor_status') {
var content = data[idx];
} else {
var content = moment(data[idx]).format('ddd MMM D HH:mm:ss ZZ YYYY')
}
$("#proofpoint_status_table > tbody").append(
$("<tr/>")
.append($("<td style='width:30%'/>").text(props[idx]))
.append($("<td/>").text(content))
);
}
}
});
});
</script>
<table class="table table-striped table-condensed" id="proofpoint_status_table">
<thead>
<tr>
<th colspan="2">
<img src="/ui/img/proofpoint.svg" class="proofpoint_logo image_invertible">
<div class="proofpoint_status_div pull-right">
<i class="fa fa-spinner fa-2x fa-spin" style="vertical-align: middle;" id="proofpoint_status"></i>
</div>
</th>
</tr>
</thead>
<tbody>
</tbody>
<tfoot>
<tr>
<td colspan="2">
<?=gettext("For more information, please visit");?>
<a target="_blank" rel="noopener noreferrer" href="https://www.deciso.com/">Deciso B.V.</a>
</td>
</tr>
</tfoot>
</table>