Not updating the guidelines in using.rst-- I don't want to encourage people to use this. now that Certbot preserves gid, the better way to set permissions is to change the group permisisons!
* Preserve other-read bit on private keys too
* fix integration test
* fix and rename permission routines in integration tests
* During the release, reset local-oldest-requirements.txts that might have been changed during development.
* only modify the file if it exists
* no need to specifically add certbot-compatibility-test because it doesn't have the file anyway
* Use double quotes instead of single
* escape dots and brackets
* Escape and quote correctly
* Remove duplicate route53 options
Duplicate route53 option (certbot-route53:auth) removed.
* Remove duplicate route53 options from help
Removed duplicate route53 options from help by adding a couple of if statements in cli.py.
* Fix formatting
* Use cleaner solution to remove duplicate route53 options from help
Used a cleaner method to remove the duplicate option from the help text
* Fix line too long
* Add regression test
Added a regression test and fixed a couple of minor issues.
* Fix trailing whitespace
* Add missing newline
* Put newline in correct place
* Rough draft of External Account Binding.
* Remove parameter --eab and namespace kid and hmac. Also add parameters to "register" subcommand.
* Refactor as much as possible of the EAB functionality into ExternalAccountBinding class.
* Remove debug line.
* Added external account binding to Directory.Meta.
* Rename to account_public_key, hmac_key and make some non-optional.
Rename command line argument to --eab-hmac-key.
* Error out when the server requires External Account Binding and the user
has not supplied kid and hmac key.
* Remove whitespace.
* Refactor a bit to make it possible to set the url argument.
* Move from_data method into client.
* Revert "Move from_data method into client."
This reverts commit 8963fae
* Refactored to use json field on Registration.
* Inherit from object according to Google Python Style Guide.
* Move to two separate ifs.
* Get tests to pass after External Account Binding additions.
* messages.py back to 100% test coverage with some EAB tests.
* .encode() this JSON key.
* Set eab parameter default values to None.
* * Remove unnecessary public key mock on most of the test.
* Restructure the directory mock to be able to mock both True and False for externalAccountRequired easily.
* Add EAB client tests.
* Move external_account_required check into BackwardsCompatibleClientV2 to be able to mock it.
* Update versions.
* Try 0.29.0.
* Revert "Try 0.29.0."
This reverts commit 5779509
* Try 0.29.0 again.
* Try this.
* Fix pylint failures.
* Add tests for external_account_required method.
* Test not needed, avoid:
************* Module acme.client_test
C: 1, 0: Too many lines in module (1258/1250) (too-many-lines)
* Move real external_account_required method into ClientV2 and pass through to it in BackwardsCompatibleClientV2.
* Handle missing meta key in server ACME directory.
* Add docstring for BackwardsCompatibleClientV2.external_account_required().
* Add tests for BackwardsCompatibleClientV2.external_account_required().
* Fix coverage for ACMEv1 code in BackwardsCompatibleClientV2.
* Disable pylint too-many-lines check for client_test.py.
* Fix versions.
* Remove whitespace that accidently snuck into an earlier commit.
* Remove these two stray whitespaces also.
* And the last couple of whitespaces.
* Add External Account Binding to changelog.
* Add dev0 suffix to setup.py.
Co-Authored-By: robaman <robert@kastel.se>
* Set to "-e acme[dev]" again.
Co-Authored-By: robaman <robert@kastel.se>
Currently the release script in master fails for a few reasons. First, it's trying to use --numprocesses which comes from a pytest plugin that we're not installing in the release script. Second, many new warnings are raised when we're not using pinned versions of our dependencies.
I'm not sure I agree, but one could argue that we should fix these issues and use the file during the release. I'm particularly hesitant for us to do this when it comes to warnings. We currently do not pin our dependencies in the release script. Do we really want to stop the release because a new package was released and is warning about something? One could argue we do because these warnings may be visible to the user, but they very rarely are and I think this makes the release process much too painful.
I especially do not think we should block the release on this now as we are not up to date on the warnings raised by the latest versions of our packages so there is a lot to work through.
* Don't use pytest.ini during the release.
* State that pytest.ini isn't used in release script.
* Setup an integration tests env against Pebble, that enforce post-as-get
* Implement POST-as-GET requests, with fallback to GET.
* Fix unit tests
* Fix coverage.
* Fix or ignore lint errors
* Corrections after review
* Correct test
* Try a simple delegate approach
* Add a test
* Simplify test mocking
* Clean comment
Fixes#1473.
writes privkey.pem to 0600 by default for new lineages
on renewals where a new privkey is generated, preserves group mode and gid
Things this PR does not do:
we talked about forcing 0600 on privkeys when a Certbot upgrade is detected. Instead, this PR only creates new lineages with the more restrictive permission to prevent renewal breakages.
this doesn't solve many of the problems mentioned in #1473 that are not directly related to the title issue!
* safe_open on archive keyfiles
* keep group from current lineage
* clean up integration test
* safe_open can follow symlinks
* fix tests on windows, maybe
* Address Brad's comments
* Revert changes to safe_open
* Test chown is called when saving new key
* Reorder chown operation
* Changelog and documentation
* Fix documentation style
* Automation for changelog changes during release
* Update changelog during release before modifying version numbers
* don't link to the GitHub repo
* no need to sign the commit bumping version numbers
* simplify tail call
Current pinned version of cffi is 1.10.0. This version does not provide pre-compiled wheels for latest Python versions on Windows. This implies on this plateform, when certbot is installed, to compile cffi from sources.
But for that, the computer will need to have the Visual C compiler available locally. This environnement is really heavy to setup, and totally outside of the scope.
This PR updates cffi to version 1.11.5, that has the required wheels, and makes certbot installable without a full .NET dev profile.
* WIP on adding a random sleep for noninteractive renewal
* Update changelog
* Log the fact that we're randomly sleeping
* stdin may better define interactivity than stdout
* Try mocking time.sleep for all tests
* Move mocked sleep elsewhere
* mock the right object
* Somewhat ugly synthetic PTY trick
* Move set -u down below self-exec
* Revert "Move set -u down below self-exec"
This reverts commit 6bde65a738.
* Revert "Somewhat ugly synthetic PTY trick"
This reverts commit 89c704a4be.
* Log specific duration of random sleep
* Test coverage for random sleep() logic in main.py
Defaults path of Certbot are the following:
config: /etc/letsencrypt
workdir: /var/letsencrypt/lib
logs: /var/letsencrypt/log
On Windows, this translate into:
config: C:\etc\letsencrypt
workdir: C:\var\letsencrypt\lib
logs: C:\var\letsencrypt\log
As Windows does not follow the standard POSIX filesystem layout, theses paths do not have a lot of sense in this case.
This PR sets the following default paths when Certbot is run on Windows:
config: C:\Certbot
workdir: C:\Certbot\lib
logs: C:\Certbot\log
Better to decide the default paths for Certbot before users start to run it on Windows, to avoid future migration procedures.
With current code, the certbot-auto self-upgrade process can make it actually to downgrade itself, because the comparison done is an equality test between local certbot-auto version and the remote one. This is a flaw for attackers, that could make certbot-auto break itself by falsely advertising it about an old version as the latest one available.
A function is added to make a more advanced comparison between version. Certbot-auto will upgrade itself only if the local version is strictly inferior to the latest one available. For instance, a version 0.28.0 will not upgrade itself if the latest one available on internet is 0.27.1. Similarly, non-official versions like 0.28.0.dev0 will never trigger a self-upgrade, to help development workflows.
This implementation relies only on the Python distribution installed by certbot-auto (supporting 2.7+) and basic shell operations, to be compatible with any UNIX-based system.
* Check version with protection again downgrade
* Create a stable version of letsencrypt-auto to use correctly self-upgrade functionality
* Update letsencrypt-auto-source/letsencrypt-auto.template
This PR has the value of VENV_NAME override any value set in the tools/venv* scripts.
I also removed the use of VENV_ARGS. This was used in _venv_common.sh as a means of passing arguments for virtualenv between the scripts, however, there is no other use of the variable in this repository and passing the arguments through a function call is much more natural in Python.
* Respect VENV_NAME in tools/venv*.
* Stop using VENV_ARGS
* Remove VENV_NAME_ENV_VAR and add docstrings.
Following some inconsistencies occurred during by developments, and in the light of #6508, it decided to wrote a PR that will take fully advantage of the conversion from bash to python to the development setup tools.
This PR adresses several issues when trying to use the development setup tools (`tools/venv.py` and `tools/venv3.py`:
* on Windows, `python` executable is not always in PATH (default behavior)
* even if the option is checked, the `python` executable is not associated to the usually symlink `python3` on Windows
* on Windows again, really powerful introspection of the available Python environments can be done with `py`, the Windows Python launcher
* in general for all systems, `tools/venv.py` and `tools/venv3.py` ensures that the respective Python major version will be used to setup the virtual environment if available.
* finally, the best and first candidate to test should be the Python executable used to launch the `tools/venv*.py` script. It was not relevant before because it was shell scripts, but do it is.
The logic is shared in `_venv_common.py`, and will be called appropriately for both scripts. In priority decreasing order, python executable will be search and tested:
* from the current Python executable, as exposed by `sys.executable`
* from any python or pythonX (X as a python version like 2, 3 or 2.7 or 3.4) executable available in PATH
* from the Windows Python launched `py` if available
Individual changes were:
* Update tools/venv3.py to support py launcher on Windows
* Fix typo in help message
* More explicit calls with space protection
* Complete refactoring to take advantage of the python runtime, and control of the compatible version to use.
After #6485 and #6435, it appears that there is no good reason to not fail fast when test, cover or linting scripts are executed.
This PR ensures to fail fast by invoking commands throught subprocess.check_call instead of subprocess.call, and by removing the handling of non-zero exit code at the end of theses scripts.
As now coverage on Windows is executed with thresholds, I added specific thresholds for this platform. Because some portions of code that are done for Unix platform will not be executed on Windows.
Note that coverage reports from Travis and AppVeyor are accumulated on Codecov. So if a file is covered up to 50 % on Linux, and all other parts are covered on Windows, then coverage is 100 % for Codecov.
Note: that PR also fixes the ability of coverage tests to fail if thresholds are exceeded.
* Use check_call to fail fast in all scripts related to tests/lint/coverage/deploy
* Make specific coverage threshold for windows
It is about the exit codes that are returned from the various scripts in tools during tox execution.
Indeed, tox relies on the non-zero exit code from a given script to know that something failed during the execution.
Previously, theses scripts were in bash, and a bash script returns an exit code that is the higher code returned from any of the command executed by the script. So if any command return a non-zero (in particular pylint or pytest), then the script return also non-zero.
Now that these scripts are converted into python, pylint and pytest are executed via subprocess, that returns the exit code as variables. But if theses codes are not handled explicitly, the python script itself will return zero if no python exception occured. As a consequence currently, Certbot CI system is unable to detect any test error or lint error, because there is no exception in this case, only exit codes from the binaries executed.
This PR fixes that, by handling correctly the exit code from the most critical scripts, install_and_test.py and tox.cover.py, but also all the scripts that I converted into Python and that could be executed in the context of a shell (via tox or directly for instance).
Certbot relies heavily on bash scripts to deploy a development environment and to execute tests. This is fine for Linux systems, including Travis, but problematic for Windows machines.
This PR converts all theses scripts into Python, to make them platform independant.
As a consequence, tox-win.ini is not needed anymore, and tox can be run indifferently on Windows or on Linux using a common tox.ini. AppVeyor is updated accordingly to execute tests for acme, certbot and all dns plugins. Other tests are not executed as they are for Docker, unsupported Apache/Nginx/Postfix plugins (for now) or not relevant for Windows (explicit Linux distribution tests or pylint).
Another PR will be done on certbot website to update how a dev environment can be set up.
* Replace several shell scripts by python equivalent.
* Correction on tox coverage
* Extend usage of new python scripts
* Various corrections
* Replace venv construction bash scripts by python equivalents
* Update tox.ini
* Unicode lines to compare files
* Put modifications on letsencrypt-auto-source instead of generated scripts
* Add executable permissions for Linux.
* Merge tox win tests into main tox
* Skip lock_test on Windows
* Correct appveyor config
* Update appveyor.yml
* Explicit coverage py27 or py37
* Avoid to cover non supported certbot plugins on Windows
* Update tox.ini
* Remove specific warnings during CI
* No cover on a debug code for tests only.
* Update documentation and help script on venv/venv3.py
* Customize help message for Windows
* Quote correctly executable path with potential spaces in it.
* Copy pipstrap from upstream
This PR update requirement of Lexicon to 2.7.14 on OVH plugin, to allow HTTP proxy to be used correctly when underlying OVH provider is invoked.
* Update Lexicon to correct use of HTTP proxy on OVH provider
* Update dev_constraints.txt
* Update CHANGELOG.md
On Linux, you can invoke os.rename(src, dst) even if dst already exists. In this case, destination file will be atomically replaced by the source file.
On Windows, this will lead to an OSError because changes are not atomic. This cause certbot renew to fail in particular, because the old certificate configuration needs to be replace by the new when a certificate is effectively renewed.
One could use the cross-platform function os.replace, but it is available only on Python >= 3.3.
This PR add a function in compat to handle correctly this case on Windows, and delegating everything else to os.rename.
* Cross platform compatible os.rename (we can use os.replace if its python 3)
* Use os.replace instead of custom non-atomic code.
* Avoid errors for lint and mypy. Add a test.
