certbot/certbot-apache/tests/centos_test.py

199 lines
8.2 KiB
Python
Raw Normal View History

Make the contents of the apache plugin private (#7579) Part of #5775. Tree: ``` certbot-apache/certbot_apache ├── __init__.py ├── _internal │   ├── apache_util.py │   ├── augeas_lens │   │   ├── httpd.aug │   │   └── README │   ├── centos-options-ssl-apache.conf │   ├── configurator.py │   ├── constants.py │   ├── display_ops.py │   ├── entrypoint.py │   ├── http_01.py │   ├── __init__.py │   ├── obj.py │   ├── options-ssl-apache.conf │   ├── override_arch.py │   ├── override_centos.py │   ├── override_darwin.py │   ├── override_debian.py │   ├── override_fedora.py │   ├── override_gentoo.py │   ├── override_suse.py │   └── parser.py └── tests ├── ... ``` * Create _internal folder for certbot_apache * Move apache_util.py to _internal * Move display_ops.py to _internal * Move override_centos.py to _internal * Move override_gentoo.py to _internal * Move override_darwin.py to _internal * Move override_suse.py to _internal * Move override_debian.py to _internal * Move override_fedora.py to _internal * Move override_arch.py to _internal * Move parser.py to _internal * Move obj.py to _internal * Move http_01.py to _internal * Move entrypoint.py to _internal * Move constants.py to _internal * Move configurator.py to _internal * Move augeas_lens to _internal * Move options-ssl-apache.conf files to _internal * move augeas_lens in MANIFEST * Clean up some stray references to certbot_apache that could use _internal * Correct imports and lint
2019-11-25 12:44:40 -05:00
"""Test for certbot_apache._internal.configurator for Centos overrides"""
Distribution specific override functionality based on class inheritance (#5202) Class inheritance based approach to distro specific overrides. How it works: The certbot-apache plugin entrypoint has been changed to entrypoint.ENTRYPOINT which is a variable containing appropriate override class for system, if available. Override classes register themselves using decorator override.register() which takes a list of distribution fingerprints (ID & LIKE variables in /etc/os-release, or platform.linux_distribution() as a fallback). These end up as keys in dict override.OVERRIDE_CLASSES and values for the keys are references to the class that called the decorator, hence allowing self-registration of override classes when they are imported. The only file importing these override classes is entrypoint.py, so adding new override classes would need only one import in addition to the actual override class file. Generic changes: Parser initialization has been moved to separate class method, allowing easy override where needed. Cleaned up configurator.py a bit, and moved some helper functions to newly created apache_util.py Split Debian specific code from configurator.py to debian_override.py Changed define_cmd to apache_cmd because the parameters are for every distribution supporting this behavior, and we're able to use the value to build the additional configuration dump commands. Moved add_parser_mod() from configurator to parser add_mod() Added two new configuration dump parsing methods to update_runtime_variables() in parser: update_includes() and update_modules(). Changed init_modules() in parser to accommodate the changes above. (ie. don't throw existing self.modules out). Moved OS based constants to their respective override classes. Refactored configurator class discovery in tests to help easier test case creation using distribution based override configurator class. tests.util.get_apache_configurator() now takes keyword argument os_info which is string of the desired mock OS fingerprint response that's used for picking the right override class. This PR includes two major generic additions that should vastly improve our parsing accuracy and quality: Includes are parsed from config dump from httpd binary. This is mandatory for some distributions (Like OpenSUSE) to get visibility over the whole configuration tree because of Include statements passed on in command line, and not via root httpd.conf file. Modules are parsed from config dump from httpd binary. This lets us jump into correct IfModule directives if for some reason we have missed the module availability (because of one being included on command line or such). Distribution specific changes Because of the generic changes, there are two distributions (or distribution families) that do not provide such functionality, so it had to be overridden in their respective override files. These distributions are: CentOS, because it deliberately limits httpd binary stdout using SELinux as a feature. We are doing opportunistic config dumps here however, in case SELinux enforcing is off. Gentoo, because it does not provide a way to invoke httpd with command line parsed from its specific configuration file. Gentoo relies heavily on Define statements that are passed over from APACHE2_OPTS variable /etc/conf.d/apache2 file and most of the configuration in root Apache configuration are dependent on these values. Debian Moved the Debian specific parts from configurator.py to Debian specific override. CentOS Parsing of /etc/sysconfig/httpd file for additional Define statements. This could hold other parameters too, but parsing everything off it would require a full Apache lexer. For CLI parameters, I think Defines are the most common ones. This is done in addition of opportunistic parsing of httpd binary config dump. Added CentOS default Apache configuration tree for realistic test cases. Gentoo Parsing Defines from /etc/conf.d/apache2 variable APACHE2_OPTS, which holds additional Define statements to enable certain functionalities, enabling parts of the configuration in the Apache2 DOM. This is done instead of trying to parse httpd binary configuration dumps. Added default Apache configuration from Gentoo to testdata, including /etc/conf.d/apache2 file for realistic test cases. * Distribution specific override functionality based on class inheritance * Need to patch get_systemd_os_like to as travis has proper os-release * Added pydoc * Move parser initialization to a method and fix Python 3 __new__ errors * Parser changes to parse HTTPD config * Try to get modules and includes from httpd process for better visibility over the configuration * Had to disable duplicate-code because of test setup (PyCQA/pylint/issues/214) * CentOS tests and linter fixes * Gentoo override, tests and linter fixes * Mock the process call in all the tests that require it * Fix CentOS test mock * Restore reseting modules list functionality for cleanup * Move OS fingerprinting and constant mocks to parent class * Fixes requested in review * New entrypoint structure and started moving OS constants to override classes * OS constants move continued, test and linter fixes * Removed dead code * Apache compatibility test changest to reflect OS constant restructure * Test fix * Requested changes * Moved Debian specific tests to own test file * Removed decorator based override class registration in favor of entrypoint dict * Fix for update_includes for some versions of Augeas * Take fedora fix into account in tests * Review fixes
2017-12-04 14:49:18 -05:00
import unittest
try:
import mock
except ImportError: # pragma: no cover
from unittest import mock # type: ignore
Distribution specific override functionality based on class inheritance (#5202) Class inheritance based approach to distro specific overrides. How it works: The certbot-apache plugin entrypoint has been changed to entrypoint.ENTRYPOINT which is a variable containing appropriate override class for system, if available. Override classes register themselves using decorator override.register() which takes a list of distribution fingerprints (ID & LIKE variables in /etc/os-release, or platform.linux_distribution() as a fallback). These end up as keys in dict override.OVERRIDE_CLASSES and values for the keys are references to the class that called the decorator, hence allowing self-registration of override classes when they are imported. The only file importing these override classes is entrypoint.py, so adding new override classes would need only one import in addition to the actual override class file. Generic changes: Parser initialization has been moved to separate class method, allowing easy override where needed. Cleaned up configurator.py a bit, and moved some helper functions to newly created apache_util.py Split Debian specific code from configurator.py to debian_override.py Changed define_cmd to apache_cmd because the parameters are for every distribution supporting this behavior, and we're able to use the value to build the additional configuration dump commands. Moved add_parser_mod() from configurator to parser add_mod() Added two new configuration dump parsing methods to update_runtime_variables() in parser: update_includes() and update_modules(). Changed init_modules() in parser to accommodate the changes above. (ie. don't throw existing self.modules out). Moved OS based constants to their respective override classes. Refactored configurator class discovery in tests to help easier test case creation using distribution based override configurator class. tests.util.get_apache_configurator() now takes keyword argument os_info which is string of the desired mock OS fingerprint response that's used for picking the right override class. This PR includes two major generic additions that should vastly improve our parsing accuracy and quality: Includes are parsed from config dump from httpd binary. This is mandatory for some distributions (Like OpenSUSE) to get visibility over the whole configuration tree because of Include statements passed on in command line, and not via root httpd.conf file. Modules are parsed from config dump from httpd binary. This lets us jump into correct IfModule directives if for some reason we have missed the module availability (because of one being included on command line or such). Distribution specific changes Because of the generic changes, there are two distributions (or distribution families) that do not provide such functionality, so it had to be overridden in their respective override files. These distributions are: CentOS, because it deliberately limits httpd binary stdout using SELinux as a feature. We are doing opportunistic config dumps here however, in case SELinux enforcing is off. Gentoo, because it does not provide a way to invoke httpd with command line parsed from its specific configuration file. Gentoo relies heavily on Define statements that are passed over from APACHE2_OPTS variable /etc/conf.d/apache2 file and most of the configuration in root Apache configuration are dependent on these values. Debian Moved the Debian specific parts from configurator.py to Debian specific override. CentOS Parsing of /etc/sysconfig/httpd file for additional Define statements. This could hold other parameters too, but parsing everything off it would require a full Apache lexer. For CLI parameters, I think Defines are the most common ones. This is done in addition of opportunistic parsing of httpd binary config dump. Added CentOS default Apache configuration tree for realistic test cases. Gentoo Parsing Defines from /etc/conf.d/apache2 variable APACHE2_OPTS, which holds additional Define statements to enable certain functionalities, enabling parts of the configuration in the Apache2 DOM. This is done instead of trying to parse httpd binary configuration dumps. Added default Apache configuration from Gentoo to testdata, including /etc/conf.d/apache2 file for realistic test cases. * Distribution specific override functionality based on class inheritance * Need to patch get_systemd_os_like to as travis has proper os-release * Added pydoc * Move parser initialization to a method and fix Python 3 __new__ errors * Parser changes to parse HTTPD config * Try to get modules and includes from httpd process for better visibility over the configuration * Had to disable duplicate-code because of test setup (PyCQA/pylint/issues/214) * CentOS tests and linter fixes * Gentoo override, tests and linter fixes * Mock the process call in all the tests that require it * Fix CentOS test mock * Restore reseting modules list functionality for cleanup * Move OS fingerprinting and constant mocks to parent class * Fixes requested in review * New entrypoint structure and started moving OS constants to override classes * OS constants move continued, test and linter fixes * Removed dead code * Apache compatibility test changest to reflect OS constant restructure * Test fix * Requested changes * Moved Debian specific tests to own test file * Removed decorator based override class registration in favor of entrypoint dict * Fix for update_includes for some versions of Augeas * Take fedora fix into account in tests * Review fixes
2017-12-04 14:49:18 -05:00
from certbot import errors
from certbot.compat import filesystem
2019-04-12 16:32:52 -04:00
from certbot.compat import os
Make the contents of the apache plugin private (#7579) Part of #5775. Tree: ``` certbot-apache/certbot_apache ├── __init__.py ├── _internal │   ├── apache_util.py │   ├── augeas_lens │   │   ├── httpd.aug │   │   └── README │   ├── centos-options-ssl-apache.conf │   ├── configurator.py │   ├── constants.py │   ├── display_ops.py │   ├── entrypoint.py │   ├── http_01.py │   ├── __init__.py │   ├── obj.py │   ├── options-ssl-apache.conf │   ├── override_arch.py │   ├── override_centos.py │   ├── override_darwin.py │   ├── override_debian.py │   ├── override_fedora.py │   ├── override_gentoo.py │   ├── override_suse.py │   └── parser.py └── tests ├── ... ``` * Create _internal folder for certbot_apache * Move apache_util.py to _internal * Move display_ops.py to _internal * Move override_centos.py to _internal * Move override_gentoo.py to _internal * Move override_darwin.py to _internal * Move override_suse.py to _internal * Move override_debian.py to _internal * Move override_fedora.py to _internal * Move override_arch.py to _internal * Move parser.py to _internal * Move obj.py to _internal * Move http_01.py to _internal * Move entrypoint.py to _internal * Move constants.py to _internal * Move configurator.py to _internal * Move augeas_lens to _internal * Move options-ssl-apache.conf files to _internal * move augeas_lens in MANIFEST * Clean up some stray references to certbot_apache that could use _internal * Correct imports and lint
2019-11-25 12:44:40 -05:00
from certbot_apache._internal import obj
from certbot_apache._internal import override_centos
import util
Distribution specific override functionality based on class inheritance (#5202) Class inheritance based approach to distro specific overrides. How it works: The certbot-apache plugin entrypoint has been changed to entrypoint.ENTRYPOINT which is a variable containing appropriate override class for system, if available. Override classes register themselves using decorator override.register() which takes a list of distribution fingerprints (ID & LIKE variables in /etc/os-release, or platform.linux_distribution() as a fallback). These end up as keys in dict override.OVERRIDE_CLASSES and values for the keys are references to the class that called the decorator, hence allowing self-registration of override classes when they are imported. The only file importing these override classes is entrypoint.py, so adding new override classes would need only one import in addition to the actual override class file. Generic changes: Parser initialization has been moved to separate class method, allowing easy override where needed. Cleaned up configurator.py a bit, and moved some helper functions to newly created apache_util.py Split Debian specific code from configurator.py to debian_override.py Changed define_cmd to apache_cmd because the parameters are for every distribution supporting this behavior, and we're able to use the value to build the additional configuration dump commands. Moved add_parser_mod() from configurator to parser add_mod() Added two new configuration dump parsing methods to update_runtime_variables() in parser: update_includes() and update_modules(). Changed init_modules() in parser to accommodate the changes above. (ie. don't throw existing self.modules out). Moved OS based constants to their respective override classes. Refactored configurator class discovery in tests to help easier test case creation using distribution based override configurator class. tests.util.get_apache_configurator() now takes keyword argument os_info which is string of the desired mock OS fingerprint response that's used for picking the right override class. This PR includes two major generic additions that should vastly improve our parsing accuracy and quality: Includes are parsed from config dump from httpd binary. This is mandatory for some distributions (Like OpenSUSE) to get visibility over the whole configuration tree because of Include statements passed on in command line, and not via root httpd.conf file. Modules are parsed from config dump from httpd binary. This lets us jump into correct IfModule directives if for some reason we have missed the module availability (because of one being included on command line or such). Distribution specific changes Because of the generic changes, there are two distributions (or distribution families) that do not provide such functionality, so it had to be overridden in their respective override files. These distributions are: CentOS, because it deliberately limits httpd binary stdout using SELinux as a feature. We are doing opportunistic config dumps here however, in case SELinux enforcing is off. Gentoo, because it does not provide a way to invoke httpd with command line parsed from its specific configuration file. Gentoo relies heavily on Define statements that are passed over from APACHE2_OPTS variable /etc/conf.d/apache2 file and most of the configuration in root Apache configuration are dependent on these values. Debian Moved the Debian specific parts from configurator.py to Debian specific override. CentOS Parsing of /etc/sysconfig/httpd file for additional Define statements. This could hold other parameters too, but parsing everything off it would require a full Apache lexer. For CLI parameters, I think Defines are the most common ones. This is done in addition of opportunistic parsing of httpd binary config dump. Added CentOS default Apache configuration tree for realistic test cases. Gentoo Parsing Defines from /etc/conf.d/apache2 variable APACHE2_OPTS, which holds additional Define statements to enable certain functionalities, enabling parts of the configuration in the Apache2 DOM. This is done instead of trying to parse httpd binary configuration dumps. Added default Apache configuration from Gentoo to testdata, including /etc/conf.d/apache2 file for realistic test cases. * Distribution specific override functionality based on class inheritance * Need to patch get_systemd_os_like to as travis has proper os-release * Added pydoc * Move parser initialization to a method and fix Python 3 __new__ errors * Parser changes to parse HTTPD config * Try to get modules and includes from httpd process for better visibility over the configuration * Had to disable duplicate-code because of test setup (PyCQA/pylint/issues/214) * CentOS tests and linter fixes * Gentoo override, tests and linter fixes * Mock the process call in all the tests that require it * Fix CentOS test mock * Restore reseting modules list functionality for cleanup * Move OS fingerprinting and constant mocks to parent class * Fixes requested in review * New entrypoint structure and started moving OS constants to override classes * OS constants move continued, test and linter fixes * Removed dead code * Apache compatibility test changest to reflect OS constant restructure * Test fix * Requested changes * Moved Debian specific tests to own test file * Removed decorator based override class registration in favor of entrypoint dict * Fix for update_includes for some versions of Augeas * Take fedora fix into account in tests * Review fixes
2017-12-04 14:49:18 -05:00
2019-04-12 16:32:52 -04:00
Distribution specific override functionality based on class inheritance (#5202) Class inheritance based approach to distro specific overrides. How it works: The certbot-apache plugin entrypoint has been changed to entrypoint.ENTRYPOINT which is a variable containing appropriate override class for system, if available. Override classes register themselves using decorator override.register() which takes a list of distribution fingerprints (ID & LIKE variables in /etc/os-release, or platform.linux_distribution() as a fallback). These end up as keys in dict override.OVERRIDE_CLASSES and values for the keys are references to the class that called the decorator, hence allowing self-registration of override classes when they are imported. The only file importing these override classes is entrypoint.py, so adding new override classes would need only one import in addition to the actual override class file. Generic changes: Parser initialization has been moved to separate class method, allowing easy override where needed. Cleaned up configurator.py a bit, and moved some helper functions to newly created apache_util.py Split Debian specific code from configurator.py to debian_override.py Changed define_cmd to apache_cmd because the parameters are for every distribution supporting this behavior, and we're able to use the value to build the additional configuration dump commands. Moved add_parser_mod() from configurator to parser add_mod() Added two new configuration dump parsing methods to update_runtime_variables() in parser: update_includes() and update_modules(). Changed init_modules() in parser to accommodate the changes above. (ie. don't throw existing self.modules out). Moved OS based constants to their respective override classes. Refactored configurator class discovery in tests to help easier test case creation using distribution based override configurator class. tests.util.get_apache_configurator() now takes keyword argument os_info which is string of the desired mock OS fingerprint response that's used for picking the right override class. This PR includes two major generic additions that should vastly improve our parsing accuracy and quality: Includes are parsed from config dump from httpd binary. This is mandatory for some distributions (Like OpenSUSE) to get visibility over the whole configuration tree because of Include statements passed on in command line, and not via root httpd.conf file. Modules are parsed from config dump from httpd binary. This lets us jump into correct IfModule directives if for some reason we have missed the module availability (because of one being included on command line or such). Distribution specific changes Because of the generic changes, there are two distributions (or distribution families) that do not provide such functionality, so it had to be overridden in their respective override files. These distributions are: CentOS, because it deliberately limits httpd binary stdout using SELinux as a feature. We are doing opportunistic config dumps here however, in case SELinux enforcing is off. Gentoo, because it does not provide a way to invoke httpd with command line parsed from its specific configuration file. Gentoo relies heavily on Define statements that are passed over from APACHE2_OPTS variable /etc/conf.d/apache2 file and most of the configuration in root Apache configuration are dependent on these values. Debian Moved the Debian specific parts from configurator.py to Debian specific override. CentOS Parsing of /etc/sysconfig/httpd file for additional Define statements. This could hold other parameters too, but parsing everything off it would require a full Apache lexer. For CLI parameters, I think Defines are the most common ones. This is done in addition of opportunistic parsing of httpd binary config dump. Added CentOS default Apache configuration tree for realistic test cases. Gentoo Parsing Defines from /etc/conf.d/apache2 variable APACHE2_OPTS, which holds additional Define statements to enable certain functionalities, enabling parts of the configuration in the Apache2 DOM. This is done instead of trying to parse httpd binary configuration dumps. Added default Apache configuration from Gentoo to testdata, including /etc/conf.d/apache2 file for realistic test cases. * Distribution specific override functionality based on class inheritance * Need to patch get_systemd_os_like to as travis has proper os-release * Added pydoc * Move parser initialization to a method and fix Python 3 __new__ errors * Parser changes to parse HTTPD config * Try to get modules and includes from httpd process for better visibility over the configuration * Had to disable duplicate-code because of test setup (PyCQA/pylint/issues/214) * CentOS tests and linter fixes * Gentoo override, tests and linter fixes * Mock the process call in all the tests that require it * Fix CentOS test mock * Restore reseting modules list functionality for cleanup * Move OS fingerprinting and constant mocks to parent class * Fixes requested in review * New entrypoint structure and started moving OS constants to override classes * OS constants move continued, test and linter fixes * Removed dead code * Apache compatibility test changest to reflect OS constant restructure * Test fix * Requested changes * Moved Debian specific tests to own test file * Removed decorator based override class registration in favor of entrypoint dict * Fix for update_includes for some versions of Augeas * Take fedora fix into account in tests * Review fixes
2017-12-04 14:49:18 -05:00
def get_vh_truth(temp_dir, config_name):
"""Return the ground truth for the specified directory."""
prefix = os.path.join(
temp_dir, config_name, "httpd/conf.d")
aug_pre = "/files" + prefix
vh_truth = [
obj.VirtualHost(
os.path.join(prefix, "centos.example.com.conf"),
os.path.join(aug_pre, "centos.example.com.conf/VirtualHost"),
{obj.Addr.fromstring("*:80")},
Distribution specific override functionality based on class inheritance (#5202) Class inheritance based approach to distro specific overrides. How it works: The certbot-apache plugin entrypoint has been changed to entrypoint.ENTRYPOINT which is a variable containing appropriate override class for system, if available. Override classes register themselves using decorator override.register() which takes a list of distribution fingerprints (ID & LIKE variables in /etc/os-release, or platform.linux_distribution() as a fallback). These end up as keys in dict override.OVERRIDE_CLASSES and values for the keys are references to the class that called the decorator, hence allowing self-registration of override classes when they are imported. The only file importing these override classes is entrypoint.py, so adding new override classes would need only one import in addition to the actual override class file. Generic changes: Parser initialization has been moved to separate class method, allowing easy override where needed. Cleaned up configurator.py a bit, and moved some helper functions to newly created apache_util.py Split Debian specific code from configurator.py to debian_override.py Changed define_cmd to apache_cmd because the parameters are for every distribution supporting this behavior, and we're able to use the value to build the additional configuration dump commands. Moved add_parser_mod() from configurator to parser add_mod() Added two new configuration dump parsing methods to update_runtime_variables() in parser: update_includes() and update_modules(). Changed init_modules() in parser to accommodate the changes above. (ie. don't throw existing self.modules out). Moved OS based constants to their respective override classes. Refactored configurator class discovery in tests to help easier test case creation using distribution based override configurator class. tests.util.get_apache_configurator() now takes keyword argument os_info which is string of the desired mock OS fingerprint response that's used for picking the right override class. This PR includes two major generic additions that should vastly improve our parsing accuracy and quality: Includes are parsed from config dump from httpd binary. This is mandatory for some distributions (Like OpenSUSE) to get visibility over the whole configuration tree because of Include statements passed on in command line, and not via root httpd.conf file. Modules are parsed from config dump from httpd binary. This lets us jump into correct IfModule directives if for some reason we have missed the module availability (because of one being included on command line or such). Distribution specific changes Because of the generic changes, there are two distributions (or distribution families) that do not provide such functionality, so it had to be overridden in their respective override files. These distributions are: CentOS, because it deliberately limits httpd binary stdout using SELinux as a feature. We are doing opportunistic config dumps here however, in case SELinux enforcing is off. Gentoo, because it does not provide a way to invoke httpd with command line parsed from its specific configuration file. Gentoo relies heavily on Define statements that are passed over from APACHE2_OPTS variable /etc/conf.d/apache2 file and most of the configuration in root Apache configuration are dependent on these values. Debian Moved the Debian specific parts from configurator.py to Debian specific override. CentOS Parsing of /etc/sysconfig/httpd file for additional Define statements. This could hold other parameters too, but parsing everything off it would require a full Apache lexer. For CLI parameters, I think Defines are the most common ones. This is done in addition of opportunistic parsing of httpd binary config dump. Added CentOS default Apache configuration tree for realistic test cases. Gentoo Parsing Defines from /etc/conf.d/apache2 variable APACHE2_OPTS, which holds additional Define statements to enable certain functionalities, enabling parts of the configuration in the Apache2 DOM. This is done instead of trying to parse httpd binary configuration dumps. Added default Apache configuration from Gentoo to testdata, including /etc/conf.d/apache2 file for realistic test cases. * Distribution specific override functionality based on class inheritance * Need to patch get_systemd_os_like to as travis has proper os-release * Added pydoc * Move parser initialization to a method and fix Python 3 __new__ errors * Parser changes to parse HTTPD config * Try to get modules and includes from httpd process for better visibility over the configuration * Had to disable duplicate-code because of test setup (PyCQA/pylint/issues/214) * CentOS tests and linter fixes * Gentoo override, tests and linter fixes * Mock the process call in all the tests that require it * Fix CentOS test mock * Restore reseting modules list functionality for cleanup * Move OS fingerprinting and constant mocks to parent class * Fixes requested in review * New entrypoint structure and started moving OS constants to override classes * OS constants move continued, test and linter fixes * Removed dead code * Apache compatibility test changest to reflect OS constant restructure * Test fix * Requested changes * Moved Debian specific tests to own test file * Removed decorator based override class registration in favor of entrypoint dict * Fix for update_includes for some versions of Augeas * Take fedora fix into account in tests * Review fixes
2017-12-04 14:49:18 -05:00
False, True, "centos.example.com"),
obj.VirtualHost(
os.path.join(prefix, "ssl.conf"),
os.path.join(aug_pre, "ssl.conf/VirtualHost"),
{obj.Addr.fromstring("_default_:443")},
Distribution specific override functionality based on class inheritance (#5202) Class inheritance based approach to distro specific overrides. How it works: The certbot-apache plugin entrypoint has been changed to entrypoint.ENTRYPOINT which is a variable containing appropriate override class for system, if available. Override classes register themselves using decorator override.register() which takes a list of distribution fingerprints (ID & LIKE variables in /etc/os-release, or platform.linux_distribution() as a fallback). These end up as keys in dict override.OVERRIDE_CLASSES and values for the keys are references to the class that called the decorator, hence allowing self-registration of override classes when they are imported. The only file importing these override classes is entrypoint.py, so adding new override classes would need only one import in addition to the actual override class file. Generic changes: Parser initialization has been moved to separate class method, allowing easy override where needed. Cleaned up configurator.py a bit, and moved some helper functions to newly created apache_util.py Split Debian specific code from configurator.py to debian_override.py Changed define_cmd to apache_cmd because the parameters are for every distribution supporting this behavior, and we're able to use the value to build the additional configuration dump commands. Moved add_parser_mod() from configurator to parser add_mod() Added two new configuration dump parsing methods to update_runtime_variables() in parser: update_includes() and update_modules(). Changed init_modules() in parser to accommodate the changes above. (ie. don't throw existing self.modules out). Moved OS based constants to their respective override classes. Refactored configurator class discovery in tests to help easier test case creation using distribution based override configurator class. tests.util.get_apache_configurator() now takes keyword argument os_info which is string of the desired mock OS fingerprint response that's used for picking the right override class. This PR includes two major generic additions that should vastly improve our parsing accuracy and quality: Includes are parsed from config dump from httpd binary. This is mandatory for some distributions (Like OpenSUSE) to get visibility over the whole configuration tree because of Include statements passed on in command line, and not via root httpd.conf file. Modules are parsed from config dump from httpd binary. This lets us jump into correct IfModule directives if for some reason we have missed the module availability (because of one being included on command line or such). Distribution specific changes Because of the generic changes, there are two distributions (or distribution families) that do not provide such functionality, so it had to be overridden in their respective override files. These distributions are: CentOS, because it deliberately limits httpd binary stdout using SELinux as a feature. We are doing opportunistic config dumps here however, in case SELinux enforcing is off. Gentoo, because it does not provide a way to invoke httpd with command line parsed from its specific configuration file. Gentoo relies heavily on Define statements that are passed over from APACHE2_OPTS variable /etc/conf.d/apache2 file and most of the configuration in root Apache configuration are dependent on these values. Debian Moved the Debian specific parts from configurator.py to Debian specific override. CentOS Parsing of /etc/sysconfig/httpd file for additional Define statements. This could hold other parameters too, but parsing everything off it would require a full Apache lexer. For CLI parameters, I think Defines are the most common ones. This is done in addition of opportunistic parsing of httpd binary config dump. Added CentOS default Apache configuration tree for realistic test cases. Gentoo Parsing Defines from /etc/conf.d/apache2 variable APACHE2_OPTS, which holds additional Define statements to enable certain functionalities, enabling parts of the configuration in the Apache2 DOM. This is done instead of trying to parse httpd binary configuration dumps. Added default Apache configuration from Gentoo to testdata, including /etc/conf.d/apache2 file for realistic test cases. * Distribution specific override functionality based on class inheritance * Need to patch get_systemd_os_like to as travis has proper os-release * Added pydoc * Move parser initialization to a method and fix Python 3 __new__ errors * Parser changes to parse HTTPD config * Try to get modules and includes from httpd process for better visibility over the configuration * Had to disable duplicate-code because of test setup (PyCQA/pylint/issues/214) * CentOS tests and linter fixes * Gentoo override, tests and linter fixes * Mock the process call in all the tests that require it * Fix CentOS test mock * Restore reseting modules list functionality for cleanup * Move OS fingerprinting and constant mocks to parent class * Fixes requested in review * New entrypoint structure and started moving OS constants to override classes * OS constants move continued, test and linter fixes * Removed dead code * Apache compatibility test changest to reflect OS constant restructure * Test fix * Requested changes * Moved Debian specific tests to own test file * Removed decorator based override class registration in favor of entrypoint dict * Fix for update_includes for some versions of Augeas * Take fedora fix into account in tests * Review fixes
2017-12-04 14:49:18 -05:00
True, True, None)
]
return vh_truth
class FedoraRestartTest(util.ApacheTest):
"""Tests for Fedora specific self-signed certificate override"""
def setUp(self): # pylint: disable=arguments-differ
test_dir = "centos7_apache/apache"
config_root = "centos7_apache/apache/httpd"
vhost_root = "centos7_apache/apache/httpd/conf.d"
super().setUp(test_dir=test_dir,
config_root=config_root,
vhost_root=vhost_root)
self.config = util.get_apache_configurator(
self.config_path, self.vhost_path, self.config_dir, self.work_dir,
os_info="fedora_old")
self.vh_truth = get_vh_truth(
self.temp_dir, "centos7_apache/apache")
def _run_fedora_test(self):
self.assertIsInstance(self.config, override_centos.CentOSConfigurator)
with mock.patch("certbot.util.get_os_info") as mock_info:
mock_info.return_value = ["fedora", "28"]
self.config.config_test()
def test_non_fedora_error(self):
Make the contents of the apache plugin private (#7579) Part of #5775. Tree: ``` certbot-apache/certbot_apache ├── __init__.py ├── _internal │   ├── apache_util.py │   ├── augeas_lens │   │   ├── httpd.aug │   │   └── README │   ├── centos-options-ssl-apache.conf │   ├── configurator.py │   ├── constants.py │   ├── display_ops.py │   ├── entrypoint.py │   ├── http_01.py │   ├── __init__.py │   ├── obj.py │   ├── options-ssl-apache.conf │   ├── override_arch.py │   ├── override_centos.py │   ├── override_darwin.py │   ├── override_debian.py │   ├── override_fedora.py │   ├── override_gentoo.py │   ├── override_suse.py │   └── parser.py └── tests ├── ... ``` * Create _internal folder for certbot_apache * Move apache_util.py to _internal * Move display_ops.py to _internal * Move override_centos.py to _internal * Move override_gentoo.py to _internal * Move override_darwin.py to _internal * Move override_suse.py to _internal * Move override_debian.py to _internal * Move override_fedora.py to _internal * Move override_arch.py to _internal * Move parser.py to _internal * Move obj.py to _internal * Move http_01.py to _internal * Move entrypoint.py to _internal * Move constants.py to _internal * Move configurator.py to _internal * Move augeas_lens to _internal * Move options-ssl-apache.conf files to _internal * move augeas_lens in MANIFEST * Clean up some stray references to certbot_apache that could use _internal * Correct imports and lint
2019-11-25 12:44:40 -05:00
c_test = "certbot_apache._internal.configurator.ApacheConfigurator.config_test"
with mock.patch(c_test) as mock_test:
mock_test.side_effect = errors.MisconfigurationError
with mock.patch("certbot.util.get_os_info") as mock_info:
mock_info.return_value = ["not_fedora"]
self.assertRaises(errors.MisconfigurationError,
self.config.config_test)
def test_fedora_restart_error(self):
Make the contents of the apache plugin private (#7579) Part of #5775. Tree: ``` certbot-apache/certbot_apache ├── __init__.py ├── _internal │   ├── apache_util.py │   ├── augeas_lens │   │   ├── httpd.aug │   │   └── README │   ├── centos-options-ssl-apache.conf │   ├── configurator.py │   ├── constants.py │   ├── display_ops.py │   ├── entrypoint.py │   ├── http_01.py │   ├── __init__.py │   ├── obj.py │   ├── options-ssl-apache.conf │   ├── override_arch.py │   ├── override_centos.py │   ├── override_darwin.py │   ├── override_debian.py │   ├── override_fedora.py │   ├── override_gentoo.py │   ├── override_suse.py │   └── parser.py └── tests ├── ... ``` * Create _internal folder for certbot_apache * Move apache_util.py to _internal * Move display_ops.py to _internal * Move override_centos.py to _internal * Move override_gentoo.py to _internal * Move override_darwin.py to _internal * Move override_suse.py to _internal * Move override_debian.py to _internal * Move override_fedora.py to _internal * Move override_arch.py to _internal * Move parser.py to _internal * Move obj.py to _internal * Move http_01.py to _internal * Move entrypoint.py to _internal * Move constants.py to _internal * Move configurator.py to _internal * Move augeas_lens to _internal * Move options-ssl-apache.conf files to _internal * move augeas_lens in MANIFEST * Clean up some stray references to certbot_apache that could use _internal * Correct imports and lint
2019-11-25 12:44:40 -05:00
c_test = "certbot_apache._internal.configurator.ApacheConfigurator.config_test"
with mock.patch(c_test) as mock_test:
# First call raises error, second doesn't
mock_test.side_effect = [errors.MisconfigurationError, '']
with mock.patch("certbot.util.run_script") as mock_run:
mock_run.side_effect = errors.SubprocessError
self.assertRaises(errors.MisconfigurationError,
self._run_fedora_test)
def test_fedora_restart(self):
Make the contents of the apache plugin private (#7579) Part of #5775. Tree: ``` certbot-apache/certbot_apache ├── __init__.py ├── _internal │   ├── apache_util.py │   ├── augeas_lens │   │   ├── httpd.aug │   │   └── README │   ├── centos-options-ssl-apache.conf │   ├── configurator.py │   ├── constants.py │   ├── display_ops.py │   ├── entrypoint.py │   ├── http_01.py │   ├── __init__.py │   ├── obj.py │   ├── options-ssl-apache.conf │   ├── override_arch.py │   ├── override_centos.py │   ├── override_darwin.py │   ├── override_debian.py │   ├── override_fedora.py │   ├── override_gentoo.py │   ├── override_suse.py │   └── parser.py └── tests ├── ... ``` * Create _internal folder for certbot_apache * Move apache_util.py to _internal * Move display_ops.py to _internal * Move override_centos.py to _internal * Move override_gentoo.py to _internal * Move override_darwin.py to _internal * Move override_suse.py to _internal * Move override_debian.py to _internal * Move override_fedora.py to _internal * Move override_arch.py to _internal * Move parser.py to _internal * Move obj.py to _internal * Move http_01.py to _internal * Move entrypoint.py to _internal * Move constants.py to _internal * Move configurator.py to _internal * Move augeas_lens to _internal * Move options-ssl-apache.conf files to _internal * move augeas_lens in MANIFEST * Clean up some stray references to certbot_apache that could use _internal * Correct imports and lint
2019-11-25 12:44:40 -05:00
c_test = "certbot_apache._internal.configurator.ApacheConfigurator.config_test"
with mock.patch(c_test) as mock_test:
with mock.patch("certbot.util.run_script") as mock_run:
# First call raises error, second doesn't
mock_test.side_effect = [errors.MisconfigurationError, '']
self._run_fedora_test()
self.assertEqual(mock_test.call_count, 2)
self.assertEqual(mock_run.call_args[0][0],
['systemctl', 'restart', 'httpd'])
Distribution specific override functionality based on class inheritance (#5202) Class inheritance based approach to distro specific overrides. How it works: The certbot-apache plugin entrypoint has been changed to entrypoint.ENTRYPOINT which is a variable containing appropriate override class for system, if available. Override classes register themselves using decorator override.register() which takes a list of distribution fingerprints (ID & LIKE variables in /etc/os-release, or platform.linux_distribution() as a fallback). These end up as keys in dict override.OVERRIDE_CLASSES and values for the keys are references to the class that called the decorator, hence allowing self-registration of override classes when they are imported. The only file importing these override classes is entrypoint.py, so adding new override classes would need only one import in addition to the actual override class file. Generic changes: Parser initialization has been moved to separate class method, allowing easy override where needed. Cleaned up configurator.py a bit, and moved some helper functions to newly created apache_util.py Split Debian specific code from configurator.py to debian_override.py Changed define_cmd to apache_cmd because the parameters are for every distribution supporting this behavior, and we're able to use the value to build the additional configuration dump commands. Moved add_parser_mod() from configurator to parser add_mod() Added two new configuration dump parsing methods to update_runtime_variables() in parser: update_includes() and update_modules(). Changed init_modules() in parser to accommodate the changes above. (ie. don't throw existing self.modules out). Moved OS based constants to their respective override classes. Refactored configurator class discovery in tests to help easier test case creation using distribution based override configurator class. tests.util.get_apache_configurator() now takes keyword argument os_info which is string of the desired mock OS fingerprint response that's used for picking the right override class. This PR includes two major generic additions that should vastly improve our parsing accuracy and quality: Includes are parsed from config dump from httpd binary. This is mandatory for some distributions (Like OpenSUSE) to get visibility over the whole configuration tree because of Include statements passed on in command line, and not via root httpd.conf file. Modules are parsed from config dump from httpd binary. This lets us jump into correct IfModule directives if for some reason we have missed the module availability (because of one being included on command line or such). Distribution specific changes Because of the generic changes, there are two distributions (or distribution families) that do not provide such functionality, so it had to be overridden in their respective override files. These distributions are: CentOS, because it deliberately limits httpd binary stdout using SELinux as a feature. We are doing opportunistic config dumps here however, in case SELinux enforcing is off. Gentoo, because it does not provide a way to invoke httpd with command line parsed from its specific configuration file. Gentoo relies heavily on Define statements that are passed over from APACHE2_OPTS variable /etc/conf.d/apache2 file and most of the configuration in root Apache configuration are dependent on these values. Debian Moved the Debian specific parts from configurator.py to Debian specific override. CentOS Parsing of /etc/sysconfig/httpd file for additional Define statements. This could hold other parameters too, but parsing everything off it would require a full Apache lexer. For CLI parameters, I think Defines are the most common ones. This is done in addition of opportunistic parsing of httpd binary config dump. Added CentOS default Apache configuration tree for realistic test cases. Gentoo Parsing Defines from /etc/conf.d/apache2 variable APACHE2_OPTS, which holds additional Define statements to enable certain functionalities, enabling parts of the configuration in the Apache2 DOM. This is done instead of trying to parse httpd binary configuration dumps. Added default Apache configuration from Gentoo to testdata, including /etc/conf.d/apache2 file for realistic test cases. * Distribution specific override functionality based on class inheritance * Need to patch get_systemd_os_like to as travis has proper os-release * Added pydoc * Move parser initialization to a method and fix Python 3 __new__ errors * Parser changes to parse HTTPD config * Try to get modules and includes from httpd process for better visibility over the configuration * Had to disable duplicate-code because of test setup (PyCQA/pylint/issues/214) * CentOS tests and linter fixes * Gentoo override, tests and linter fixes * Mock the process call in all the tests that require it * Fix CentOS test mock * Restore reseting modules list functionality for cleanup * Move OS fingerprinting and constant mocks to parent class * Fixes requested in review * New entrypoint structure and started moving OS constants to override classes * OS constants move continued, test and linter fixes * Removed dead code * Apache compatibility test changest to reflect OS constant restructure * Test fix * Requested changes * Moved Debian specific tests to own test file * Removed decorator based override class registration in favor of entrypoint dict * Fix for update_includes for some versions of Augeas * Take fedora fix into account in tests * Review fixes
2017-12-04 14:49:18 -05:00
class MultipleVhostsTestCentOS(util.ApacheTest):
"""Multiple vhost tests for CentOS / RHEL family of distros"""
_multiprocess_can_split_ = True
def setUp(self): # pylint: disable=arguments-differ
test_dir = "centos7_apache/apache"
config_root = "centos7_apache/apache/httpd"
vhost_root = "centos7_apache/apache/httpd/conf.d"
super().setUp(test_dir=test_dir,
config_root=config_root,
vhost_root=vhost_root)
Distribution specific override functionality based on class inheritance (#5202) Class inheritance based approach to distro specific overrides. How it works: The certbot-apache plugin entrypoint has been changed to entrypoint.ENTRYPOINT which is a variable containing appropriate override class for system, if available. Override classes register themselves using decorator override.register() which takes a list of distribution fingerprints (ID & LIKE variables in /etc/os-release, or platform.linux_distribution() as a fallback). These end up as keys in dict override.OVERRIDE_CLASSES and values for the keys are references to the class that called the decorator, hence allowing self-registration of override classes when they are imported. The only file importing these override classes is entrypoint.py, so adding new override classes would need only one import in addition to the actual override class file. Generic changes: Parser initialization has been moved to separate class method, allowing easy override where needed. Cleaned up configurator.py a bit, and moved some helper functions to newly created apache_util.py Split Debian specific code from configurator.py to debian_override.py Changed define_cmd to apache_cmd because the parameters are for every distribution supporting this behavior, and we're able to use the value to build the additional configuration dump commands. Moved add_parser_mod() from configurator to parser add_mod() Added two new configuration dump parsing methods to update_runtime_variables() in parser: update_includes() and update_modules(). Changed init_modules() in parser to accommodate the changes above. (ie. don't throw existing self.modules out). Moved OS based constants to their respective override classes. Refactored configurator class discovery in tests to help easier test case creation using distribution based override configurator class. tests.util.get_apache_configurator() now takes keyword argument os_info which is string of the desired mock OS fingerprint response that's used for picking the right override class. This PR includes two major generic additions that should vastly improve our parsing accuracy and quality: Includes are parsed from config dump from httpd binary. This is mandatory for some distributions (Like OpenSUSE) to get visibility over the whole configuration tree because of Include statements passed on in command line, and not via root httpd.conf file. Modules are parsed from config dump from httpd binary. This lets us jump into correct IfModule directives if for some reason we have missed the module availability (because of one being included on command line or such). Distribution specific changes Because of the generic changes, there are two distributions (or distribution families) that do not provide such functionality, so it had to be overridden in their respective override files. These distributions are: CentOS, because it deliberately limits httpd binary stdout using SELinux as a feature. We are doing opportunistic config dumps here however, in case SELinux enforcing is off. Gentoo, because it does not provide a way to invoke httpd with command line parsed from its specific configuration file. Gentoo relies heavily on Define statements that are passed over from APACHE2_OPTS variable /etc/conf.d/apache2 file and most of the configuration in root Apache configuration are dependent on these values. Debian Moved the Debian specific parts from configurator.py to Debian specific override. CentOS Parsing of /etc/sysconfig/httpd file for additional Define statements. This could hold other parameters too, but parsing everything off it would require a full Apache lexer. For CLI parameters, I think Defines are the most common ones. This is done in addition of opportunistic parsing of httpd binary config dump. Added CentOS default Apache configuration tree for realistic test cases. Gentoo Parsing Defines from /etc/conf.d/apache2 variable APACHE2_OPTS, which holds additional Define statements to enable certain functionalities, enabling parts of the configuration in the Apache2 DOM. This is done instead of trying to parse httpd binary configuration dumps. Added default Apache configuration from Gentoo to testdata, including /etc/conf.d/apache2 file for realistic test cases. * Distribution specific override functionality based on class inheritance * Need to patch get_systemd_os_like to as travis has proper os-release * Added pydoc * Move parser initialization to a method and fix Python 3 __new__ errors * Parser changes to parse HTTPD config * Try to get modules and includes from httpd process for better visibility over the configuration * Had to disable duplicate-code because of test setup (PyCQA/pylint/issues/214) * CentOS tests and linter fixes * Gentoo override, tests and linter fixes * Mock the process call in all the tests that require it * Fix CentOS test mock * Restore reseting modules list functionality for cleanup * Move OS fingerprinting and constant mocks to parent class * Fixes requested in review * New entrypoint structure and started moving OS constants to override classes * OS constants move continued, test and linter fixes * Removed dead code * Apache compatibility test changest to reflect OS constant restructure * Test fix * Requested changes * Moved Debian specific tests to own test file * Removed decorator based override class registration in favor of entrypoint dict * Fix for update_includes for some versions of Augeas * Take fedora fix into account in tests * Review fixes
2017-12-04 14:49:18 -05:00
self.config = util.get_apache_configurator(
self.config_path, self.vhost_path, self.config_dir, self.work_dir,
os_info="centos")
self.vh_truth = get_vh_truth(
self.temp_dir, "centos7_apache/apache")
def test_get_parser(self):
self.assertIsInstance(self.config.parser, override_centos.CentOSParser)
Distribution specific override functionality based on class inheritance (#5202) Class inheritance based approach to distro specific overrides. How it works: The certbot-apache plugin entrypoint has been changed to entrypoint.ENTRYPOINT which is a variable containing appropriate override class for system, if available. Override classes register themselves using decorator override.register() which takes a list of distribution fingerprints (ID & LIKE variables in /etc/os-release, or platform.linux_distribution() as a fallback). These end up as keys in dict override.OVERRIDE_CLASSES and values for the keys are references to the class that called the decorator, hence allowing self-registration of override classes when they are imported. The only file importing these override classes is entrypoint.py, so adding new override classes would need only one import in addition to the actual override class file. Generic changes: Parser initialization has been moved to separate class method, allowing easy override where needed. Cleaned up configurator.py a bit, and moved some helper functions to newly created apache_util.py Split Debian specific code from configurator.py to debian_override.py Changed define_cmd to apache_cmd because the parameters are for every distribution supporting this behavior, and we're able to use the value to build the additional configuration dump commands. Moved add_parser_mod() from configurator to parser add_mod() Added two new configuration dump parsing methods to update_runtime_variables() in parser: update_includes() and update_modules(). Changed init_modules() in parser to accommodate the changes above. (ie. don't throw existing self.modules out). Moved OS based constants to their respective override classes. Refactored configurator class discovery in tests to help easier test case creation using distribution based override configurator class. tests.util.get_apache_configurator() now takes keyword argument os_info which is string of the desired mock OS fingerprint response that's used for picking the right override class. This PR includes two major generic additions that should vastly improve our parsing accuracy and quality: Includes are parsed from config dump from httpd binary. This is mandatory for some distributions (Like OpenSUSE) to get visibility over the whole configuration tree because of Include statements passed on in command line, and not via root httpd.conf file. Modules are parsed from config dump from httpd binary. This lets us jump into correct IfModule directives if for some reason we have missed the module availability (because of one being included on command line or such). Distribution specific changes Because of the generic changes, there are two distributions (or distribution families) that do not provide such functionality, so it had to be overridden in their respective override files. These distributions are: CentOS, because it deliberately limits httpd binary stdout using SELinux as a feature. We are doing opportunistic config dumps here however, in case SELinux enforcing is off. Gentoo, because it does not provide a way to invoke httpd with command line parsed from its specific configuration file. Gentoo relies heavily on Define statements that are passed over from APACHE2_OPTS variable /etc/conf.d/apache2 file and most of the configuration in root Apache configuration are dependent on these values. Debian Moved the Debian specific parts from configurator.py to Debian specific override. CentOS Parsing of /etc/sysconfig/httpd file for additional Define statements. This could hold other parameters too, but parsing everything off it would require a full Apache lexer. For CLI parameters, I think Defines are the most common ones. This is done in addition of opportunistic parsing of httpd binary config dump. Added CentOS default Apache configuration tree for realistic test cases. Gentoo Parsing Defines from /etc/conf.d/apache2 variable APACHE2_OPTS, which holds additional Define statements to enable certain functionalities, enabling parts of the configuration in the Apache2 DOM. This is done instead of trying to parse httpd binary configuration dumps. Added default Apache configuration from Gentoo to testdata, including /etc/conf.d/apache2 file for realistic test cases. * Distribution specific override functionality based on class inheritance * Need to patch get_systemd_os_like to as travis has proper os-release * Added pydoc * Move parser initialization to a method and fix Python 3 __new__ errors * Parser changes to parse HTTPD config * Try to get modules and includes from httpd process for better visibility over the configuration * Had to disable duplicate-code because of test setup (PyCQA/pylint/issues/214) * CentOS tests and linter fixes * Gentoo override, tests and linter fixes * Mock the process call in all the tests that require it * Fix CentOS test mock * Restore reseting modules list functionality for cleanup * Move OS fingerprinting and constant mocks to parent class * Fixes requested in review * New entrypoint structure and started moving OS constants to override classes * OS constants move continued, test and linter fixes * Removed dead code * Apache compatibility test changest to reflect OS constant restructure * Test fix * Requested changes * Moved Debian specific tests to own test file * Removed decorator based override class registration in favor of entrypoint dict * Fix for update_includes for some versions of Augeas * Take fedora fix into account in tests * Review fixes
2017-12-04 14:49:18 -05:00
@mock.patch("certbot_apache._internal.apache_util._get_runtime_cfg")
Distribution specific override functionality based on class inheritance (#5202) Class inheritance based approach to distro specific overrides. How it works: The certbot-apache plugin entrypoint has been changed to entrypoint.ENTRYPOINT which is a variable containing appropriate override class for system, if available. Override classes register themselves using decorator override.register() which takes a list of distribution fingerprints (ID & LIKE variables in /etc/os-release, or platform.linux_distribution() as a fallback). These end up as keys in dict override.OVERRIDE_CLASSES and values for the keys are references to the class that called the decorator, hence allowing self-registration of override classes when they are imported. The only file importing these override classes is entrypoint.py, so adding new override classes would need only one import in addition to the actual override class file. Generic changes: Parser initialization has been moved to separate class method, allowing easy override where needed. Cleaned up configurator.py a bit, and moved some helper functions to newly created apache_util.py Split Debian specific code from configurator.py to debian_override.py Changed define_cmd to apache_cmd because the parameters are for every distribution supporting this behavior, and we're able to use the value to build the additional configuration dump commands. Moved add_parser_mod() from configurator to parser add_mod() Added two new configuration dump parsing methods to update_runtime_variables() in parser: update_includes() and update_modules(). Changed init_modules() in parser to accommodate the changes above. (ie. don't throw existing self.modules out). Moved OS based constants to their respective override classes. Refactored configurator class discovery in tests to help easier test case creation using distribution based override configurator class. tests.util.get_apache_configurator() now takes keyword argument os_info which is string of the desired mock OS fingerprint response that's used for picking the right override class. This PR includes two major generic additions that should vastly improve our parsing accuracy and quality: Includes are parsed from config dump from httpd binary. This is mandatory for some distributions (Like OpenSUSE) to get visibility over the whole configuration tree because of Include statements passed on in command line, and not via root httpd.conf file. Modules are parsed from config dump from httpd binary. This lets us jump into correct IfModule directives if for some reason we have missed the module availability (because of one being included on command line or such). Distribution specific changes Because of the generic changes, there are two distributions (or distribution families) that do not provide such functionality, so it had to be overridden in their respective override files. These distributions are: CentOS, because it deliberately limits httpd binary stdout using SELinux as a feature. We are doing opportunistic config dumps here however, in case SELinux enforcing is off. Gentoo, because it does not provide a way to invoke httpd with command line parsed from its specific configuration file. Gentoo relies heavily on Define statements that are passed over from APACHE2_OPTS variable /etc/conf.d/apache2 file and most of the configuration in root Apache configuration are dependent on these values. Debian Moved the Debian specific parts from configurator.py to Debian specific override. CentOS Parsing of /etc/sysconfig/httpd file for additional Define statements. This could hold other parameters too, but parsing everything off it would require a full Apache lexer. For CLI parameters, I think Defines are the most common ones. This is done in addition of opportunistic parsing of httpd binary config dump. Added CentOS default Apache configuration tree for realistic test cases. Gentoo Parsing Defines from /etc/conf.d/apache2 variable APACHE2_OPTS, which holds additional Define statements to enable certain functionalities, enabling parts of the configuration in the Apache2 DOM. This is done instead of trying to parse httpd binary configuration dumps. Added default Apache configuration from Gentoo to testdata, including /etc/conf.d/apache2 file for realistic test cases. * Distribution specific override functionality based on class inheritance * Need to patch get_systemd_os_like to as travis has proper os-release * Added pydoc * Move parser initialization to a method and fix Python 3 __new__ errors * Parser changes to parse HTTPD config * Try to get modules and includes from httpd process for better visibility over the configuration * Had to disable duplicate-code because of test setup (PyCQA/pylint/issues/214) * CentOS tests and linter fixes * Gentoo override, tests and linter fixes * Mock the process call in all the tests that require it * Fix CentOS test mock * Restore reseting modules list functionality for cleanup * Move OS fingerprinting and constant mocks to parent class * Fixes requested in review * New entrypoint structure and started moving OS constants to override classes * OS constants move continued, test and linter fixes * Removed dead code * Apache compatibility test changest to reflect OS constant restructure * Test fix * Requested changes * Moved Debian specific tests to own test file * Removed decorator based override class registration in favor of entrypoint dict * Fix for update_includes for some versions of Augeas * Take fedora fix into account in tests * Review fixes
2017-12-04 14:49:18 -05:00
def test_opportunistic_httpd_runtime_parsing(self, mock_get):
define_val = (
'Define: TEST1\n'
'Define: TEST2\n'
'Define: DUMP_RUN_CFG\n'
)
mod_val = (
'Loaded Modules:\n'
' mock_module (static)\n'
' another_module (static)\n'
)
def mock_get_cfg(command):
"""Mock httpd process stdout"""
if command == ['apachectl', '-t', '-D', 'DUMP_RUN_CFG']:
return define_val
elif command == ['apachectl', '-t', '-D', 'DUMP_MODULES']:
return mod_val
return ""
mock_get.side_effect = mock_get_cfg
Disable TLS session tickets in Apache (#7771) Fixes #7350. This PR changes the parsed modules from a `set` to a `dict`, with the filepath argument as the value. Accordingly, after calling `enable_mod` to enable `ssl_module`, modules now need to be re-parsed, so call `reset_modules`. * Add mechanism for selecting apache config file, based on work done in #7191. * Check OpenSSL version * Remove os imports * debian override still needs os * Reformat remaining apache tests with modules dict syntax * Clean up more apache tests * Switch from property to method for openssl and add tests for coverage. * Sometimes the dict location will be None in which case we should in fact return None * warn thoroughly and consistently in openssl_version function * update tests for new warnings * read file as bytes, and factor out the open for testing * normalize ssl_module_location path to account for being relative to server root * Use byte literals in a python 2 and 3 compatible way * string does need to be a literal * patch builtins open * add debug, remove space * Add test to check if OpenSSL detection is working on different systems * fix relative test location for cwd * put </IfModule> on its own line in test case * Revert test file to status in master. * Call augeas load before reparsing modules to pick up the changes * fix grep, tail, and mod_ssl location on centos * strip the trailing whitespace from fedora * just use LooseVersion in test * call apache2ctl on debian systems * Use sudo for apache2ctl command * add check to make sure we're getting a version * Add boolean so we don't warn on debian/ubuntu before trying to enable mod_ssl * Reduce warnings while testing by setting mock _openssl_version. * Make sure we're not throwing away any unwritten changes to the config * test last warning case for coverage * text changes for clarity
2020-03-23 19:49:52 -04:00
self.config.parser.modules = {}
Distribution specific override functionality based on class inheritance (#5202) Class inheritance based approach to distro specific overrides. How it works: The certbot-apache plugin entrypoint has been changed to entrypoint.ENTRYPOINT which is a variable containing appropriate override class for system, if available. Override classes register themselves using decorator override.register() which takes a list of distribution fingerprints (ID & LIKE variables in /etc/os-release, or platform.linux_distribution() as a fallback). These end up as keys in dict override.OVERRIDE_CLASSES and values for the keys are references to the class that called the decorator, hence allowing self-registration of override classes when they are imported. The only file importing these override classes is entrypoint.py, so adding new override classes would need only one import in addition to the actual override class file. Generic changes: Parser initialization has been moved to separate class method, allowing easy override where needed. Cleaned up configurator.py a bit, and moved some helper functions to newly created apache_util.py Split Debian specific code from configurator.py to debian_override.py Changed define_cmd to apache_cmd because the parameters are for every distribution supporting this behavior, and we're able to use the value to build the additional configuration dump commands. Moved add_parser_mod() from configurator to parser add_mod() Added two new configuration dump parsing methods to update_runtime_variables() in parser: update_includes() and update_modules(). Changed init_modules() in parser to accommodate the changes above. (ie. don't throw existing self.modules out). Moved OS based constants to their respective override classes. Refactored configurator class discovery in tests to help easier test case creation using distribution based override configurator class. tests.util.get_apache_configurator() now takes keyword argument os_info which is string of the desired mock OS fingerprint response that's used for picking the right override class. This PR includes two major generic additions that should vastly improve our parsing accuracy and quality: Includes are parsed from config dump from httpd binary. This is mandatory for some distributions (Like OpenSUSE) to get visibility over the whole configuration tree because of Include statements passed on in command line, and not via root httpd.conf file. Modules are parsed from config dump from httpd binary. This lets us jump into correct IfModule directives if for some reason we have missed the module availability (because of one being included on command line or such). Distribution specific changes Because of the generic changes, there are two distributions (or distribution families) that do not provide such functionality, so it had to be overridden in their respective override files. These distributions are: CentOS, because it deliberately limits httpd binary stdout using SELinux as a feature. We are doing opportunistic config dumps here however, in case SELinux enforcing is off. Gentoo, because it does not provide a way to invoke httpd with command line parsed from its specific configuration file. Gentoo relies heavily on Define statements that are passed over from APACHE2_OPTS variable /etc/conf.d/apache2 file and most of the configuration in root Apache configuration are dependent on these values. Debian Moved the Debian specific parts from configurator.py to Debian specific override. CentOS Parsing of /etc/sysconfig/httpd file for additional Define statements. This could hold other parameters too, but parsing everything off it would require a full Apache lexer. For CLI parameters, I think Defines are the most common ones. This is done in addition of opportunistic parsing of httpd binary config dump. Added CentOS default Apache configuration tree for realistic test cases. Gentoo Parsing Defines from /etc/conf.d/apache2 variable APACHE2_OPTS, which holds additional Define statements to enable certain functionalities, enabling parts of the configuration in the Apache2 DOM. This is done instead of trying to parse httpd binary configuration dumps. Added default Apache configuration from Gentoo to testdata, including /etc/conf.d/apache2 file for realistic test cases. * Distribution specific override functionality based on class inheritance * Need to patch get_systemd_os_like to as travis has proper os-release * Added pydoc * Move parser initialization to a method and fix Python 3 __new__ errors * Parser changes to parse HTTPD config * Try to get modules and includes from httpd process for better visibility over the configuration * Had to disable duplicate-code because of test setup (PyCQA/pylint/issues/214) * CentOS tests and linter fixes * Gentoo override, tests and linter fixes * Mock the process call in all the tests that require it * Fix CentOS test mock * Restore reseting modules list functionality for cleanup * Move OS fingerprinting and constant mocks to parent class * Fixes requested in review * New entrypoint structure and started moving OS constants to override classes * OS constants move continued, test and linter fixes * Removed dead code * Apache compatibility test changest to reflect OS constant restructure * Test fix * Requested changes * Moved Debian specific tests to own test file * Removed decorator based override class registration in favor of entrypoint dict * Fix for update_includes for some versions of Augeas * Take fedora fix into account in tests * Review fixes
2017-12-04 14:49:18 -05:00
self.config.parser.variables = {}
with mock.patch("certbot.util.get_os_info") as mock_osi:
# Make sure we have the have the CentOS httpd constants
mock_osi.return_value = ("centos", "7")
self.config.parser.update_runtime_variables()
2018-07-11 20:33:04 -04:00
self.assertEqual(mock_get.call_count, 3)
self.assertEqual(len(self.config.parser.modules), 4)
self.assertEqual(len(self.config.parser.variables), 2)
self.assertIn("TEST2", self.config.parser.variables)
self.assertIn("mod_another.c", self.config.parser.modules)
Distribution specific override functionality based on class inheritance (#5202) Class inheritance based approach to distro specific overrides. How it works: The certbot-apache plugin entrypoint has been changed to entrypoint.ENTRYPOINT which is a variable containing appropriate override class for system, if available. Override classes register themselves using decorator override.register() which takes a list of distribution fingerprints (ID & LIKE variables in /etc/os-release, or platform.linux_distribution() as a fallback). These end up as keys in dict override.OVERRIDE_CLASSES and values for the keys are references to the class that called the decorator, hence allowing self-registration of override classes when they are imported. The only file importing these override classes is entrypoint.py, so adding new override classes would need only one import in addition to the actual override class file. Generic changes: Parser initialization has been moved to separate class method, allowing easy override where needed. Cleaned up configurator.py a bit, and moved some helper functions to newly created apache_util.py Split Debian specific code from configurator.py to debian_override.py Changed define_cmd to apache_cmd because the parameters are for every distribution supporting this behavior, and we're able to use the value to build the additional configuration dump commands. Moved add_parser_mod() from configurator to parser add_mod() Added two new configuration dump parsing methods to update_runtime_variables() in parser: update_includes() and update_modules(). Changed init_modules() in parser to accommodate the changes above. (ie. don't throw existing self.modules out). Moved OS based constants to their respective override classes. Refactored configurator class discovery in tests to help easier test case creation using distribution based override configurator class. tests.util.get_apache_configurator() now takes keyword argument os_info which is string of the desired mock OS fingerprint response that's used for picking the right override class. This PR includes two major generic additions that should vastly improve our parsing accuracy and quality: Includes are parsed from config dump from httpd binary. This is mandatory for some distributions (Like OpenSUSE) to get visibility over the whole configuration tree because of Include statements passed on in command line, and not via root httpd.conf file. Modules are parsed from config dump from httpd binary. This lets us jump into correct IfModule directives if for some reason we have missed the module availability (because of one being included on command line or such). Distribution specific changes Because of the generic changes, there are two distributions (or distribution families) that do not provide such functionality, so it had to be overridden in their respective override files. These distributions are: CentOS, because it deliberately limits httpd binary stdout using SELinux as a feature. We are doing opportunistic config dumps here however, in case SELinux enforcing is off. Gentoo, because it does not provide a way to invoke httpd with command line parsed from its specific configuration file. Gentoo relies heavily on Define statements that are passed over from APACHE2_OPTS variable /etc/conf.d/apache2 file and most of the configuration in root Apache configuration are dependent on these values. Debian Moved the Debian specific parts from configurator.py to Debian specific override. CentOS Parsing of /etc/sysconfig/httpd file for additional Define statements. This could hold other parameters too, but parsing everything off it would require a full Apache lexer. For CLI parameters, I think Defines are the most common ones. This is done in addition of opportunistic parsing of httpd binary config dump. Added CentOS default Apache configuration tree for realistic test cases. Gentoo Parsing Defines from /etc/conf.d/apache2 variable APACHE2_OPTS, which holds additional Define statements to enable certain functionalities, enabling parts of the configuration in the Apache2 DOM. This is done instead of trying to parse httpd binary configuration dumps. Added default Apache configuration from Gentoo to testdata, including /etc/conf.d/apache2 file for realistic test cases. * Distribution specific override functionality based on class inheritance * Need to patch get_systemd_os_like to as travis has proper os-release * Added pydoc * Move parser initialization to a method and fix Python 3 __new__ errors * Parser changes to parse HTTPD config * Try to get modules and includes from httpd process for better visibility over the configuration * Had to disable duplicate-code because of test setup (PyCQA/pylint/issues/214) * CentOS tests and linter fixes * Gentoo override, tests and linter fixes * Mock the process call in all the tests that require it * Fix CentOS test mock * Restore reseting modules list functionality for cleanup * Move OS fingerprinting and constant mocks to parent class * Fixes requested in review * New entrypoint structure and started moving OS constants to override classes * OS constants move continued, test and linter fixes * Removed dead code * Apache compatibility test changest to reflect OS constant restructure * Test fix * Requested changes * Moved Debian specific tests to own test file * Removed decorator based override class registration in favor of entrypoint dict * Fix for update_includes for some versions of Augeas * Take fedora fix into account in tests * Review fixes
2017-12-04 14:49:18 -05:00
def test_get_virtual_hosts(self):
"""Make sure all vhosts are being properly found."""
vhs = self.config.get_virtual_hosts()
self.assertEqual(len(vhs), 2)
found = 0
for vhost in vhs:
for centos_truth in self.vh_truth:
if vhost == centos_truth:
found += 1
break
else:
raise Exception("Missed: %s" % vhost) # pragma: no cover
self.assertEqual(found, 2)
@mock.patch("certbot_apache._internal.apache_util._get_runtime_cfg")
Distribution specific override functionality based on class inheritance (#5202) Class inheritance based approach to distro specific overrides. How it works: The certbot-apache plugin entrypoint has been changed to entrypoint.ENTRYPOINT which is a variable containing appropriate override class for system, if available. Override classes register themselves using decorator override.register() which takes a list of distribution fingerprints (ID & LIKE variables in /etc/os-release, or platform.linux_distribution() as a fallback). These end up as keys in dict override.OVERRIDE_CLASSES and values for the keys are references to the class that called the decorator, hence allowing self-registration of override classes when they are imported. The only file importing these override classes is entrypoint.py, so adding new override classes would need only one import in addition to the actual override class file. Generic changes: Parser initialization has been moved to separate class method, allowing easy override where needed. Cleaned up configurator.py a bit, and moved some helper functions to newly created apache_util.py Split Debian specific code from configurator.py to debian_override.py Changed define_cmd to apache_cmd because the parameters are for every distribution supporting this behavior, and we're able to use the value to build the additional configuration dump commands. Moved add_parser_mod() from configurator to parser add_mod() Added two new configuration dump parsing methods to update_runtime_variables() in parser: update_includes() and update_modules(). Changed init_modules() in parser to accommodate the changes above. (ie. don't throw existing self.modules out). Moved OS based constants to their respective override classes. Refactored configurator class discovery in tests to help easier test case creation using distribution based override configurator class. tests.util.get_apache_configurator() now takes keyword argument os_info which is string of the desired mock OS fingerprint response that's used for picking the right override class. This PR includes two major generic additions that should vastly improve our parsing accuracy and quality: Includes are parsed from config dump from httpd binary. This is mandatory for some distributions (Like OpenSUSE) to get visibility over the whole configuration tree because of Include statements passed on in command line, and not via root httpd.conf file. Modules are parsed from config dump from httpd binary. This lets us jump into correct IfModule directives if for some reason we have missed the module availability (because of one being included on command line or such). Distribution specific changes Because of the generic changes, there are two distributions (or distribution families) that do not provide such functionality, so it had to be overridden in their respective override files. These distributions are: CentOS, because it deliberately limits httpd binary stdout using SELinux as a feature. We are doing opportunistic config dumps here however, in case SELinux enforcing is off. Gentoo, because it does not provide a way to invoke httpd with command line parsed from its specific configuration file. Gentoo relies heavily on Define statements that are passed over from APACHE2_OPTS variable /etc/conf.d/apache2 file and most of the configuration in root Apache configuration are dependent on these values. Debian Moved the Debian specific parts from configurator.py to Debian specific override. CentOS Parsing of /etc/sysconfig/httpd file for additional Define statements. This could hold other parameters too, but parsing everything off it would require a full Apache lexer. For CLI parameters, I think Defines are the most common ones. This is done in addition of opportunistic parsing of httpd binary config dump. Added CentOS default Apache configuration tree for realistic test cases. Gentoo Parsing Defines from /etc/conf.d/apache2 variable APACHE2_OPTS, which holds additional Define statements to enable certain functionalities, enabling parts of the configuration in the Apache2 DOM. This is done instead of trying to parse httpd binary configuration dumps. Added default Apache configuration from Gentoo to testdata, including /etc/conf.d/apache2 file for realistic test cases. * Distribution specific override functionality based on class inheritance * Need to patch get_systemd_os_like to as travis has proper os-release * Added pydoc * Move parser initialization to a method and fix Python 3 __new__ errors * Parser changes to parse HTTPD config * Try to get modules and includes from httpd process for better visibility over the configuration * Had to disable duplicate-code because of test setup (PyCQA/pylint/issues/214) * CentOS tests and linter fixes * Gentoo override, tests and linter fixes * Mock the process call in all the tests that require it * Fix CentOS test mock * Restore reseting modules list functionality for cleanup * Move OS fingerprinting and constant mocks to parent class * Fixes requested in review * New entrypoint structure and started moving OS constants to override classes * OS constants move continued, test and linter fixes * Removed dead code * Apache compatibility test changest to reflect OS constant restructure * Test fix * Requested changes * Moved Debian specific tests to own test file * Removed decorator based override class registration in favor of entrypoint dict * Fix for update_includes for some versions of Augeas * Take fedora fix into account in tests * Review fixes
2017-12-04 14:49:18 -05:00
def test_get_sysconfig_vars(self, mock_cfg):
"""Make sure we read the sysconfig OPTIONS variable correctly"""
# Return nothing for the process calls
mock_cfg.return_value = ""
self.config.parser.sysconfig_filep = filesystem.realpath(
Distribution specific override functionality based on class inheritance (#5202) Class inheritance based approach to distro specific overrides. How it works: The certbot-apache plugin entrypoint has been changed to entrypoint.ENTRYPOINT which is a variable containing appropriate override class for system, if available. Override classes register themselves using decorator override.register() which takes a list of distribution fingerprints (ID & LIKE variables in /etc/os-release, or platform.linux_distribution() as a fallback). These end up as keys in dict override.OVERRIDE_CLASSES and values for the keys are references to the class that called the decorator, hence allowing self-registration of override classes when they are imported. The only file importing these override classes is entrypoint.py, so adding new override classes would need only one import in addition to the actual override class file. Generic changes: Parser initialization has been moved to separate class method, allowing easy override where needed. Cleaned up configurator.py a bit, and moved some helper functions to newly created apache_util.py Split Debian specific code from configurator.py to debian_override.py Changed define_cmd to apache_cmd because the parameters are for every distribution supporting this behavior, and we're able to use the value to build the additional configuration dump commands. Moved add_parser_mod() from configurator to parser add_mod() Added two new configuration dump parsing methods to update_runtime_variables() in parser: update_includes() and update_modules(). Changed init_modules() in parser to accommodate the changes above. (ie. don't throw existing self.modules out). Moved OS based constants to their respective override classes. Refactored configurator class discovery in tests to help easier test case creation using distribution based override configurator class. tests.util.get_apache_configurator() now takes keyword argument os_info which is string of the desired mock OS fingerprint response that's used for picking the right override class. This PR includes two major generic additions that should vastly improve our parsing accuracy and quality: Includes are parsed from config dump from httpd binary. This is mandatory for some distributions (Like OpenSUSE) to get visibility over the whole configuration tree because of Include statements passed on in command line, and not via root httpd.conf file. Modules are parsed from config dump from httpd binary. This lets us jump into correct IfModule directives if for some reason we have missed the module availability (because of one being included on command line or such). Distribution specific changes Because of the generic changes, there are two distributions (or distribution families) that do not provide such functionality, so it had to be overridden in their respective override files. These distributions are: CentOS, because it deliberately limits httpd binary stdout using SELinux as a feature. We are doing opportunistic config dumps here however, in case SELinux enforcing is off. Gentoo, because it does not provide a way to invoke httpd with command line parsed from its specific configuration file. Gentoo relies heavily on Define statements that are passed over from APACHE2_OPTS variable /etc/conf.d/apache2 file and most of the configuration in root Apache configuration are dependent on these values. Debian Moved the Debian specific parts from configurator.py to Debian specific override. CentOS Parsing of /etc/sysconfig/httpd file for additional Define statements. This could hold other parameters too, but parsing everything off it would require a full Apache lexer. For CLI parameters, I think Defines are the most common ones. This is done in addition of opportunistic parsing of httpd binary config dump. Added CentOS default Apache configuration tree for realistic test cases. Gentoo Parsing Defines from /etc/conf.d/apache2 variable APACHE2_OPTS, which holds additional Define statements to enable certain functionalities, enabling parts of the configuration in the Apache2 DOM. This is done instead of trying to parse httpd binary configuration dumps. Added default Apache configuration from Gentoo to testdata, including /etc/conf.d/apache2 file for realistic test cases. * Distribution specific override functionality based on class inheritance * Need to patch get_systemd_os_like to as travis has proper os-release * Added pydoc * Move parser initialization to a method and fix Python 3 __new__ errors * Parser changes to parse HTTPD config * Try to get modules and includes from httpd process for better visibility over the configuration * Had to disable duplicate-code because of test setup (PyCQA/pylint/issues/214) * CentOS tests and linter fixes * Gentoo override, tests and linter fixes * Mock the process call in all the tests that require it * Fix CentOS test mock * Restore reseting modules list functionality for cleanup * Move OS fingerprinting and constant mocks to parent class * Fixes requested in review * New entrypoint structure and started moving OS constants to override classes * OS constants move continued, test and linter fixes * Removed dead code * Apache compatibility test changest to reflect OS constant restructure * Test fix * Requested changes * Moved Debian specific tests to own test file * Removed decorator based override class registration in favor of entrypoint dict * Fix for update_includes for some versions of Augeas * Take fedora fix into account in tests * Review fixes
2017-12-04 14:49:18 -05:00
os.path.join(self.config.parser.root, "../sysconfig/httpd"))
self.config.parser.variables = {}
with mock.patch("certbot.util.get_os_info") as mock_osi:
# Make sure we have the have the CentOS httpd constants
mock_osi.return_value = ("centos", "7")
self.config.parser.update_runtime_variables()
self.assertIn("mock_define", self.config.parser.variables)
self.assertIn("mock_define_too", self.config.parser.variables)
self.assertIn("mock_value", self.config.parser.variables)
Distribution specific override functionality based on class inheritance (#5202) Class inheritance based approach to distro specific overrides. How it works: The certbot-apache plugin entrypoint has been changed to entrypoint.ENTRYPOINT which is a variable containing appropriate override class for system, if available. Override classes register themselves using decorator override.register() which takes a list of distribution fingerprints (ID & LIKE variables in /etc/os-release, or platform.linux_distribution() as a fallback). These end up as keys in dict override.OVERRIDE_CLASSES and values for the keys are references to the class that called the decorator, hence allowing self-registration of override classes when they are imported. The only file importing these override classes is entrypoint.py, so adding new override classes would need only one import in addition to the actual override class file. Generic changes: Parser initialization has been moved to separate class method, allowing easy override where needed. Cleaned up configurator.py a bit, and moved some helper functions to newly created apache_util.py Split Debian specific code from configurator.py to debian_override.py Changed define_cmd to apache_cmd because the parameters are for every distribution supporting this behavior, and we're able to use the value to build the additional configuration dump commands. Moved add_parser_mod() from configurator to parser add_mod() Added two new configuration dump parsing methods to update_runtime_variables() in parser: update_includes() and update_modules(). Changed init_modules() in parser to accommodate the changes above. (ie. don't throw existing self.modules out). Moved OS based constants to their respective override classes. Refactored configurator class discovery in tests to help easier test case creation using distribution based override configurator class. tests.util.get_apache_configurator() now takes keyword argument os_info which is string of the desired mock OS fingerprint response that's used for picking the right override class. This PR includes two major generic additions that should vastly improve our parsing accuracy and quality: Includes are parsed from config dump from httpd binary. This is mandatory for some distributions (Like OpenSUSE) to get visibility over the whole configuration tree because of Include statements passed on in command line, and not via root httpd.conf file. Modules are parsed from config dump from httpd binary. This lets us jump into correct IfModule directives if for some reason we have missed the module availability (because of one being included on command line or such). Distribution specific changes Because of the generic changes, there are two distributions (or distribution families) that do not provide such functionality, so it had to be overridden in their respective override files. These distributions are: CentOS, because it deliberately limits httpd binary stdout using SELinux as a feature. We are doing opportunistic config dumps here however, in case SELinux enforcing is off. Gentoo, because it does not provide a way to invoke httpd with command line parsed from its specific configuration file. Gentoo relies heavily on Define statements that are passed over from APACHE2_OPTS variable /etc/conf.d/apache2 file and most of the configuration in root Apache configuration are dependent on these values. Debian Moved the Debian specific parts from configurator.py to Debian specific override. CentOS Parsing of /etc/sysconfig/httpd file for additional Define statements. This could hold other parameters too, but parsing everything off it would require a full Apache lexer. For CLI parameters, I think Defines are the most common ones. This is done in addition of opportunistic parsing of httpd binary config dump. Added CentOS default Apache configuration tree for realistic test cases. Gentoo Parsing Defines from /etc/conf.d/apache2 variable APACHE2_OPTS, which holds additional Define statements to enable certain functionalities, enabling parts of the configuration in the Apache2 DOM. This is done instead of trying to parse httpd binary configuration dumps. Added default Apache configuration from Gentoo to testdata, including /etc/conf.d/apache2 file for realistic test cases. * Distribution specific override functionality based on class inheritance * Need to patch get_systemd_os_like to as travis has proper os-release * Added pydoc * Move parser initialization to a method and fix Python 3 __new__ errors * Parser changes to parse HTTPD config * Try to get modules and includes from httpd process for better visibility over the configuration * Had to disable duplicate-code because of test setup (PyCQA/pylint/issues/214) * CentOS tests and linter fixes * Gentoo override, tests and linter fixes * Mock the process call in all the tests that require it * Fix CentOS test mock * Restore reseting modules list functionality for cleanup * Move OS fingerprinting and constant mocks to parent class * Fixes requested in review * New entrypoint structure and started moving OS constants to override classes * OS constants move continued, test and linter fixes * Removed dead code * Apache compatibility test changest to reflect OS constant restructure * Test fix * Requested changes * Moved Debian specific tests to own test file * Removed decorator based override class registration in favor of entrypoint dict * Fix for update_includes for some versions of Augeas * Take fedora fix into account in tests * Review fixes
2017-12-04 14:49:18 -05:00
self.assertEqual("TRUE", self.config.parser.variables["mock_value"])
self.assertIn("MOCK_NOSEP", self.config.parser.variables)
self.assertEqual("NOSEP_VAL", self.config.parser.variables["NOSEP_TWO"])
Distribution specific override functionality based on class inheritance (#5202) Class inheritance based approach to distro specific overrides. How it works: The certbot-apache plugin entrypoint has been changed to entrypoint.ENTRYPOINT which is a variable containing appropriate override class for system, if available. Override classes register themselves using decorator override.register() which takes a list of distribution fingerprints (ID & LIKE variables in /etc/os-release, or platform.linux_distribution() as a fallback). These end up as keys in dict override.OVERRIDE_CLASSES and values for the keys are references to the class that called the decorator, hence allowing self-registration of override classes when they are imported. The only file importing these override classes is entrypoint.py, so adding new override classes would need only one import in addition to the actual override class file. Generic changes: Parser initialization has been moved to separate class method, allowing easy override where needed. Cleaned up configurator.py a bit, and moved some helper functions to newly created apache_util.py Split Debian specific code from configurator.py to debian_override.py Changed define_cmd to apache_cmd because the parameters are for every distribution supporting this behavior, and we're able to use the value to build the additional configuration dump commands. Moved add_parser_mod() from configurator to parser add_mod() Added two new configuration dump parsing methods to update_runtime_variables() in parser: update_includes() and update_modules(). Changed init_modules() in parser to accommodate the changes above. (ie. don't throw existing self.modules out). Moved OS based constants to their respective override classes. Refactored configurator class discovery in tests to help easier test case creation using distribution based override configurator class. tests.util.get_apache_configurator() now takes keyword argument os_info which is string of the desired mock OS fingerprint response that's used for picking the right override class. This PR includes two major generic additions that should vastly improve our parsing accuracy and quality: Includes are parsed from config dump from httpd binary. This is mandatory for some distributions (Like OpenSUSE) to get visibility over the whole configuration tree because of Include statements passed on in command line, and not via root httpd.conf file. Modules are parsed from config dump from httpd binary. This lets us jump into correct IfModule directives if for some reason we have missed the module availability (because of one being included on command line or such). Distribution specific changes Because of the generic changes, there are two distributions (or distribution families) that do not provide such functionality, so it had to be overridden in their respective override files. These distributions are: CentOS, because it deliberately limits httpd binary stdout using SELinux as a feature. We are doing opportunistic config dumps here however, in case SELinux enforcing is off. Gentoo, because it does not provide a way to invoke httpd with command line parsed from its specific configuration file. Gentoo relies heavily on Define statements that are passed over from APACHE2_OPTS variable /etc/conf.d/apache2 file and most of the configuration in root Apache configuration are dependent on these values. Debian Moved the Debian specific parts from configurator.py to Debian specific override. CentOS Parsing of /etc/sysconfig/httpd file for additional Define statements. This could hold other parameters too, but parsing everything off it would require a full Apache lexer. For CLI parameters, I think Defines are the most common ones. This is done in addition of opportunistic parsing of httpd binary config dump. Added CentOS default Apache configuration tree for realistic test cases. Gentoo Parsing Defines from /etc/conf.d/apache2 variable APACHE2_OPTS, which holds additional Define statements to enable certain functionalities, enabling parts of the configuration in the Apache2 DOM. This is done instead of trying to parse httpd binary configuration dumps. Added default Apache configuration from Gentoo to testdata, including /etc/conf.d/apache2 file for realistic test cases. * Distribution specific override functionality based on class inheritance * Need to patch get_systemd_os_like to as travis has proper os-release * Added pydoc * Move parser initialization to a method and fix Python 3 __new__ errors * Parser changes to parse HTTPD config * Try to get modules and includes from httpd process for better visibility over the configuration * Had to disable duplicate-code because of test setup (PyCQA/pylint/issues/214) * CentOS tests and linter fixes * Gentoo override, tests and linter fixes * Mock the process call in all the tests that require it * Fix CentOS test mock * Restore reseting modules list functionality for cleanup * Move OS fingerprinting and constant mocks to parent class * Fixes requested in review * New entrypoint structure and started moving OS constants to override classes * OS constants move continued, test and linter fixes * Removed dead code * Apache compatibility test changest to reflect OS constant restructure * Test fix * Requested changes * Moved Debian specific tests to own test file * Removed decorator based override class registration in favor of entrypoint dict * Fix for update_includes for some versions of Augeas * Take fedora fix into account in tests * Review fixes
2017-12-04 14:49:18 -05:00
Make the contents of the apache plugin private (#7579) Part of #5775. Tree: ``` certbot-apache/certbot_apache ├── __init__.py ├── _internal │   ├── apache_util.py │   ├── augeas_lens │   │   ├── httpd.aug │   │   └── README │   ├── centos-options-ssl-apache.conf │   ├── configurator.py │   ├── constants.py │   ├── display_ops.py │   ├── entrypoint.py │   ├── http_01.py │   ├── __init__.py │   ├── obj.py │   ├── options-ssl-apache.conf │   ├── override_arch.py │   ├── override_centos.py │   ├── override_darwin.py │   ├── override_debian.py │   ├── override_fedora.py │   ├── override_gentoo.py │   ├── override_suse.py │   └── parser.py └── tests ├── ... ``` * Create _internal folder for certbot_apache * Move apache_util.py to _internal * Move display_ops.py to _internal * Move override_centos.py to _internal * Move override_gentoo.py to _internal * Move override_darwin.py to _internal * Move override_suse.py to _internal * Move override_debian.py to _internal * Move override_fedora.py to _internal * Move override_arch.py to _internal * Move parser.py to _internal * Move obj.py to _internal * Move http_01.py to _internal * Move entrypoint.py to _internal * Move constants.py to _internal * Move configurator.py to _internal * Move augeas_lens to _internal * Move options-ssl-apache.conf files to _internal * move augeas_lens in MANIFEST * Clean up some stray references to certbot_apache that could use _internal * Correct imports and lint
2019-11-25 12:44:40 -05:00
@mock.patch("certbot_apache._internal.configurator.util.run_script")
def test_alt_restart_works(self, mock_run_script):
mock_run_script.side_effect = [None, errors.SubprocessError, None]
self.config.restart()
2018-07-11 20:33:04 -04:00
self.assertEqual(mock_run_script.call_count, 3)
Make the contents of the apache plugin private (#7579) Part of #5775. Tree: ``` certbot-apache/certbot_apache ├── __init__.py ├── _internal │   ├── apache_util.py │   ├── augeas_lens │   │   ├── httpd.aug │   │   └── README │   ├── centos-options-ssl-apache.conf │   ├── configurator.py │   ├── constants.py │   ├── display_ops.py │   ├── entrypoint.py │   ├── http_01.py │   ├── __init__.py │   ├── obj.py │   ├── options-ssl-apache.conf │   ├── override_arch.py │   ├── override_centos.py │   ├── override_darwin.py │   ├── override_debian.py │   ├── override_fedora.py │   ├── override_gentoo.py │   ├── override_suse.py │   └── parser.py └── tests ├── ... ``` * Create _internal folder for certbot_apache * Move apache_util.py to _internal * Move display_ops.py to _internal * Move override_centos.py to _internal * Move override_gentoo.py to _internal * Move override_darwin.py to _internal * Move override_suse.py to _internal * Move override_debian.py to _internal * Move override_fedora.py to _internal * Move override_arch.py to _internal * Move parser.py to _internal * Move obj.py to _internal * Move http_01.py to _internal * Move entrypoint.py to _internal * Move constants.py to _internal * Move configurator.py to _internal * Move augeas_lens to _internal * Move options-ssl-apache.conf files to _internal * move augeas_lens in MANIFEST * Clean up some stray references to certbot_apache that could use _internal * Correct imports and lint
2019-11-25 12:44:40 -05:00
@mock.patch("certbot_apache._internal.configurator.util.run_script")
def test_alt_restart_errors(self, mock_run_script):
mock_run_script.side_effect = [None,
errors.SubprocessError,
errors.SubprocessError]
self.assertRaises(errors.MisconfigurationError, self.config.restart)
Distribution specific override functionality based on class inheritance (#5202) Class inheritance based approach to distro specific overrides. How it works: The certbot-apache plugin entrypoint has been changed to entrypoint.ENTRYPOINT which is a variable containing appropriate override class for system, if available. Override classes register themselves using decorator override.register() which takes a list of distribution fingerprints (ID & LIKE variables in /etc/os-release, or platform.linux_distribution() as a fallback). These end up as keys in dict override.OVERRIDE_CLASSES and values for the keys are references to the class that called the decorator, hence allowing self-registration of override classes when they are imported. The only file importing these override classes is entrypoint.py, so adding new override classes would need only one import in addition to the actual override class file. Generic changes: Parser initialization has been moved to separate class method, allowing easy override where needed. Cleaned up configurator.py a bit, and moved some helper functions to newly created apache_util.py Split Debian specific code from configurator.py to debian_override.py Changed define_cmd to apache_cmd because the parameters are for every distribution supporting this behavior, and we're able to use the value to build the additional configuration dump commands. Moved add_parser_mod() from configurator to parser add_mod() Added two new configuration dump parsing methods to update_runtime_variables() in parser: update_includes() and update_modules(). Changed init_modules() in parser to accommodate the changes above. (ie. don't throw existing self.modules out). Moved OS based constants to their respective override classes. Refactored configurator class discovery in tests to help easier test case creation using distribution based override configurator class. tests.util.get_apache_configurator() now takes keyword argument os_info which is string of the desired mock OS fingerprint response that's used for picking the right override class. This PR includes two major generic additions that should vastly improve our parsing accuracy and quality: Includes are parsed from config dump from httpd binary. This is mandatory for some distributions (Like OpenSUSE) to get visibility over the whole configuration tree because of Include statements passed on in command line, and not via root httpd.conf file. Modules are parsed from config dump from httpd binary. This lets us jump into correct IfModule directives if for some reason we have missed the module availability (because of one being included on command line or such). Distribution specific changes Because of the generic changes, there are two distributions (or distribution families) that do not provide such functionality, so it had to be overridden in their respective override files. These distributions are: CentOS, because it deliberately limits httpd binary stdout using SELinux as a feature. We are doing opportunistic config dumps here however, in case SELinux enforcing is off. Gentoo, because it does not provide a way to invoke httpd with command line parsed from its specific configuration file. Gentoo relies heavily on Define statements that are passed over from APACHE2_OPTS variable /etc/conf.d/apache2 file and most of the configuration in root Apache configuration are dependent on these values. Debian Moved the Debian specific parts from configurator.py to Debian specific override. CentOS Parsing of /etc/sysconfig/httpd file for additional Define statements. This could hold other parameters too, but parsing everything off it would require a full Apache lexer. For CLI parameters, I think Defines are the most common ones. This is done in addition of opportunistic parsing of httpd binary config dump. Added CentOS default Apache configuration tree for realistic test cases. Gentoo Parsing Defines from /etc/conf.d/apache2 variable APACHE2_OPTS, which holds additional Define statements to enable certain functionalities, enabling parts of the configuration in the Apache2 DOM. This is done instead of trying to parse httpd binary configuration dumps. Added default Apache configuration from Gentoo to testdata, including /etc/conf.d/apache2 file for realistic test cases. * Distribution specific override functionality based on class inheritance * Need to patch get_systemd_os_like to as travis has proper os-release * Added pydoc * Move parser initialization to a method and fix Python 3 __new__ errors * Parser changes to parse HTTPD config * Try to get modules and includes from httpd process for better visibility over the configuration * Had to disable duplicate-code because of test setup (PyCQA/pylint/issues/214) * CentOS tests and linter fixes * Gentoo override, tests and linter fixes * Mock the process call in all the tests that require it * Fix CentOS test mock * Restore reseting modules list functionality for cleanup * Move OS fingerprinting and constant mocks to parent class * Fixes requested in review * New entrypoint structure and started moving OS constants to override classes * OS constants move continued, test and linter fixes * Removed dead code * Apache compatibility test changest to reflect OS constant restructure * Test fix * Requested changes * Moved Debian specific tests to own test file * Removed decorator based override class registration in favor of entrypoint dict * Fix for update_includes for some versions of Augeas * Take fedora fix into account in tests * Review fixes
2017-12-04 14:49:18 -05:00
if __name__ == "__main__":
unittest.main() # pragma: no cover