vault/command/server/config.go

1531 lines
44 KiB
Go
Raw Normal View History

// Copyright IBM Corp. 2016, 2025
// SPDX-License-Identifier: BUSL-1.1
2015-03-12 18:21:11 -04:00
package server
import (
"encoding/json"
"errors"
2015-03-12 18:21:11 -04:00
"fmt"
"io"
"math"
2015-03-12 18:21:11 -04:00
"os"
"path/filepath"
VAULT-33008: ipv6: always display RFC-5952 §4 conformant addresses (#29228) USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for IPv6-Only capabilities. This section requires that whenever Vault displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that _all_ IPv6 addresses must conform to RFC-5952 §4 text representation recommendations[2]. These recommendations do not prevent us from accepting RFC-4241[3] IPv6 addresses, however, whenever these same addresses are displayed they must conform to the strict RFC-5952 §4 guidelines. This PR implements handling of IPv6 address conformance in our `vault server` routine. We handle conformance normalization for all server, http_proxy, listener, seal, storage and telemetry configuration where an input could contain an IPv6 address, whether configured via an HCL file or via corresponding environment variables. The approach I've taken is to handle conformance normalization at parse time to ensure that all log output and subsequent usage inside of Vaults various subsystems always reference a conformant address, that way we don't need concern ourselves with conformance later. This approach ought to be backwards compatible to prior loose address configuration requirements, with the understanding that going forward all IPv6 representation will be strict regardless of what has been configured. In many cases I've updated our various parser functions to call the new `configutil.NormalizeAddr()` to apply conformance normalization. Others required no changes because they rely on standard library URL string output, which always displays IPv6 URLs in a conformant way. Not included in this changes is any other vault exec mode other than server. Client, operator commands, agent mode, proxy mode, etc. will be included in subsequent changes if necessary. [0]: https://www.nist.gov/publications/usgv6-profile [1]: https://www.nist.gov/publications/nist-ipv6-profile [2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4 [3]: https://www.rfc-editor.org/rfc/rfc4291 Signed-off-by: Ryan Cragun <me@ryan.ec>
2025-01-27 16:14:28 -05:00
"slices"
"strconv"
2015-03-12 18:21:11 -04:00
"strings"
2015-07-30 13:21:41 -04:00
"time"
2015-03-12 18:21:11 -04:00
"github.com/hashicorp/go-discover"
"github.com/hashicorp/go-multierror"
"github.com/hashicorp/go-secure-stdlib/parseutil"
2015-03-12 18:21:11 -04:00
"github.com/hashicorp/hcl"
"github.com/hashicorp/hcl/hcl/ast"
"github.com/hashicorp/vault/helper/experiments"
"github.com/hashicorp/vault/helper/osutil"
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
"github.com/hashicorp/vault/helper/random"
"github.com/hashicorp/vault/internalshared/configutil"
"github.com/hashicorp/vault/sdk/helper/consts"
"github.com/hashicorp/vault/sdk/helper/strutil"
"github.com/hashicorp/vault/sdk/helper/testcluster"
"github.com/hashicorp/vault/vault/observations"
"github.com/mitchellh/mapstructure"
2015-03-12 18:21:11 -04:00
)
const (
VaultDevCAFilename = "vault-ca.pem"
VaultDevCertFilename = "vault-cert.pem"
VaultDevKeyFilename = "vault-key.pem"
)
// Modified internally for testing.
var validExperiments = experiments.ValidExperiments()
2015-03-12 18:21:11 -04:00
// Config is the configuration for the vault server.
type Config struct {
UnusedKeys configutil.UnusedKeyMap `hcl:",unusedKeyPositions"`
FoundKeys []string `hcl:",decodedFields"`
2020-02-14 19:39:13 -05:00
entConfig
*configutil.SharedConfig `hcl:"-"`
2015-04-17 14:25:20 -04:00
Storage *Storage `hcl:"-"`
HAStorage *Storage `hcl:"-"`
ServiceRegistration *ServiceRegistration `hcl:"-"`
2017-02-24 10:45:29 -05:00
Experiments []string `hcl:"experiments"`
EnableUnauthenticatedAccess []string `hcl:"enable_unauthenticated_access"`
CacheSize int `hcl:"cache_size"`
DisableCache bool `hcl:"-"`
DisableCacheRaw interface{} `hcl:"disable_cache"`
DisablePrintableCheck bool `hcl:"-"`
DisablePrintableCheckRaw interface{} `hcl:"disable_printable_check"`
2015-07-14 18:27:18 -04:00
EnableUI bool `hcl:"-"`
EnableUIRaw interface{} `hcl:"ui"`
2017-02-24 10:45:29 -05:00
MaxLeaseTTL time.Duration `hcl:"-"`
MaxLeaseTTLRaw interface{} `hcl:"max_lease_ttl,alias:MaxLeaseTTL"`
DefaultLeaseTTL time.Duration `hcl:"-"`
DefaultLeaseTTLRaw interface{} `hcl:"default_lease_ttl,alias:DefaultLeaseTTL"`
RemoveIrrevocableLeaseAfter time.Duration `hcl:"-"`
RemoveIrrevocableLeaseAfterRaw interface{} `hcl:"remove_irrevocable_lease_after,alias:RemoveIrrevocableLeaseAfter"`
ClusterCipherSuites string `hcl:"cluster_cipher_suites"`
2017-04-03 20:52:29 -04:00
PluginDirectory string `hcl:"plugin_directory"`
PluginTmpdir string `hcl:"plugin_tmpdir"`
PluginFileUid int `hcl:"plugin_file_uid"`
PluginFilePermissions int `hcl:"-"`
PluginFilePermissionsRaw interface{} `hcl:"plugin_file_permissions,alias:PluginFilePermissions"`
EnableIntrospectionEndpoint bool `hcl:"-"`
EnableIntrospectionEndpointRaw interface{} `hcl:"introspection_endpoint,alias:EnableIntrospectionEndpoint"`
EnableRawEndpoint bool `hcl:"-"`
EnableRawEndpointRaw interface{} `hcl:"raw_storage_endpoint,alias:EnableRawEndpoint"`
APIAddr string `hcl:"api_addr"`
ClusterAddr string `hcl:"cluster_addr"`
DisableClustering bool `hcl:"-"`
DisableClusteringRaw interface{} `hcl:"disable_clustering,alias:DisableClustering"`
2018-05-30 08:34:27 -04:00
AllowAuditLogPrefixing bool `hcl:"-"`
AllowAuditLogPrefixingRaw interface{} `hcl:"allow_audit_log_prefixing,alias:AllowAuditLogPrefixing"`
2018-08-24 12:09:03 -04:00
DisablePerformanceStandby bool `hcl:"-"`
DisablePerformanceStandbyRaw interface{} `hcl:"disable_performance_standby,alias:DisablePerformanceStandby"`
2018-08-24 12:09:03 -04:00
2018-05-30 08:34:27 -04:00
DisableSealWrap bool `hcl:"-"`
DisableSealWrapRaw interface{} `hcl:"disable_sealwrap,alias:DisableSealWrap"`
2018-10-23 15:03:17 -04:00
2018-10-23 15:09:35 -04:00
DisableIndexing bool `hcl:"-"`
DisableIndexingRaw interface{} `hcl:"disable_indexing,alias:DisableIndexing"`
DisableSentinelTrace bool `hcl:"-"`
DisableSentinelTraceRaw interface{} `hcl:"disable_sentinel_trace,alias:DisableSentinelTrace"`
EnableResponseHeaderHostname bool `hcl:"-"`
EnableResponseHeaderHostnameRaw interface{} `hcl:"enable_response_header_hostname"`
LogRequestsLevel string `hcl:"-"`
LogRequestsLevelRaw interface{} `hcl:"log_requests_level"`
DetectDeadlocks string `hcl:"detect_deadlocks"`
Observations *observations.ObservationSystemConfig `hcl:"observations"`
ImpreciseLeaseRoleTracking bool `hcl:"imprecise_lease_role_tracking"`
EnableResponseHeaderRaftNodeID bool `hcl:"-"`
EnableResponseHeaderRaftNodeIDRaw interface{} `hcl:"enable_response_header_raft_node_id"`
License string `hcl:"-"`
LicensePath string `hcl:"license_path"`
DisableSSCTokens bool `hcl:"-"`
EnablePostUnsealTrace bool `hcl:"enable_post_unseal_trace"`
PostUnsealTraceDir string `hcl:"post_unseal_trace_directory"`
ReportingScanDirectory string `hcl:"reporting_scan_directory"`
2015-03-12 18:21:11 -04:00
}
const (
sectionSeal = "Seal"
)
func (c *Config) Validate(sourceFilePath string) []configutil.ConfigError {
results := configutil.ValidateUnusedFields(c.UnusedKeys, sourceFilePath)
if c.Telemetry != nil {
results = append(results, c.Telemetry.Validate(sourceFilePath)...)
}
if c.ServiceRegistration != nil {
results = append(results, c.ServiceRegistration.Validate(sourceFilePath)...)
}
for _, l := range c.Listeners {
results = append(results, l.Validate(sourceFilePath)...)
}
results = append(results, entValidateConfig(c, sourceFilePath)...)
return results
}
2015-03-31 19:44:47 -04:00
// DevConfig is a Config that is used for dev mode of Vault.
func DevConfig(storageType string) (*Config, error) {
hclStr := `
disable_mlock = true
listener "tcp" {
address = "127.0.0.1:8200"
tls_disable = true
proxy_protocol_behavior = "allow_authorized"
proxy_protocol_authorized_addrs = "127.0.0.1:8200"
2015-03-31 19:44:47 -04:00
}
telemetry {
prometheus_retention_time = "24h"
disable_hostname = true
2015-03-12 18:21:11 -04:00
}
enable_raw_endpoint = true
2015-03-12 18:21:11 -04:00
storage "%s" {
}
2020-07-02 18:18:57 -04:00
ui = true
`
hclStr = fmt.Sprintf(hclStr, storageType)
parsed, err := ParseConfig(hclStr, "")
if err != nil {
return nil, fmt.Errorf("error parsing dev config: %w", err)
}
return parsed, nil
}
// DevTLSConfig is a Config that is used for dev tls mode of Vault.
func DevTLSConfig(storageType, certDir string, extraSANs []string) (*Config, error) {
ca, err := GenerateCA()
if err != nil {
return nil, err
}
cert, key, err := generateCert(ca.Template, ca.Signer, extraSANs)
if err != nil {
return nil, err
}
if err := os.WriteFile(fmt.Sprintf("%s/%s", certDir, VaultDevCAFilename), []byte(ca.PEM), 0o444); err != nil {
return nil, err
}
if err := os.WriteFile(fmt.Sprintf("%s/%s", certDir, VaultDevCertFilename), []byte(cert), 0o400); err != nil {
return nil, err
}
if err := os.WriteFile(fmt.Sprintf("%s/%s", certDir, VaultDevKeyFilename), []byte(key), 0o400); err != nil {
return nil, err
}
return parseDevTLSConfig(storageType, certDir)
}
func parseDevTLSConfig(storageType, certDir string) (*Config, error) {
hclStr := `
disable_mlock = true
listener "tcp" {
address = "[::]:8200"
tls_cert_file = "%s/vault-cert.pem"
tls_key_file = "%s/vault-key.pem"
proxy_protocol_behavior = "allow_authorized"
proxy_protocol_authorized_addrs = "[::]:8200"
}
telemetry {
prometheus_retention_time = "24h"
disable_hostname = true
}
enable_raw_endpoint = true
storage "%s" {
}
ui = true
`
certDirEscaped := strings.Replace(certDir, "\\", "\\\\", -1)
hclStr = fmt.Sprintf(hclStr, certDirEscaped, certDirEscaped, storageType)
parsed, err := ParseConfig(hclStr, "")
if err != nil {
return nil, err
}
return parsed, nil
}
// Storage is the underlying storage configuration for the server.
type Storage struct {
Type string
RedirectAddr string
ClusterAddr string
DisableClustering bool
Config map[string]string
2015-03-12 18:21:11 -04:00
}
func (b *Storage) GoString() string {
2015-03-12 18:21:11 -04:00
return fmt.Sprintf("*%#v", *b)
}
// ServiceRegistration is the optional service discovery for the server.
type ServiceRegistration struct {
UnusedKeys configutil.UnusedKeyMap `hcl:",unusedKeyPositions"`
Type string
Config map[string]string
}
func (b *ServiceRegistration) Validate(source string) []configutil.ConfigError {
return configutil.ValidateUnusedFields(b.UnusedKeys, source)
}
func (b *ServiceRegistration) GoString() string {
return fmt.Sprintf("*%#v", *b)
}
func NewConfig() *Config {
return &Config{
SharedConfig: new(configutil.SharedConfig),
}
2015-07-14 18:27:18 -04:00
}
2015-03-12 18:21:11 -04:00
// Merge merges two configurations.
func (c *Config) Merge(c2 *Config) *Config {
if c2 == nil {
return c
}
result := NewConfig()
result.SharedConfig = c.SharedConfig
if c2.SharedConfig != nil {
result.SharedConfig = c.SharedConfig.Merge(c2.SharedConfig)
2015-03-12 18:21:11 -04:00
}
result.Storage = c.Storage
if c2.Storage != nil {
result.Storage = c2.Storage
2015-03-12 18:21:11 -04:00
}
result.HAStorage = c.HAStorage
if c2.HAStorage != nil {
result.HAStorage = c2.HAStorage
2016-03-21 16:56:13 -04:00
}
result.ServiceRegistration = c.ServiceRegistration
if c2.ServiceRegistration != nil {
result.ServiceRegistration = c2.ServiceRegistration
}
result.CacheSize = c.CacheSize
if c2.CacheSize != 0 {
result.CacheSize = c2.CacheSize
}
// merging these booleans via an OR operation
result.DisableCache = c.DisableCache
if c2.DisableCache {
result.DisableCache = c2.DisableCache
}
result.DisableSentinelTrace = c.DisableSentinelTrace
if c2.DisableSentinelTrace {
result.DisableSentinelTrace = c2.DisableSentinelTrace
}
result.DisablePrintableCheck = c.DisablePrintableCheck
if c2.DisablePrintableCheckRaw != nil {
result.DisablePrintableCheck = c2.DisablePrintableCheck
}
// merge these integers via a MAX operation
result.MaxLeaseTTL = c.MaxLeaseTTL
if c2.MaxLeaseTTL > result.MaxLeaseTTL {
result.MaxLeaseTTL = c2.MaxLeaseTTL
}
result.DefaultLeaseTTL = c.DefaultLeaseTTL
if c2.DefaultLeaseTTL > result.DefaultLeaseTTL {
result.DefaultLeaseTTL = c2.DefaultLeaseTTL
}
result.RemoveIrrevocableLeaseAfter = c.RemoveIrrevocableLeaseAfter
if c2.RemoveIrrevocableLeaseAfter > result.RemoveIrrevocableLeaseAfter {
result.RemoveIrrevocableLeaseAfter = c2.RemoveIrrevocableLeaseAfter
}
result.ClusterCipherSuites = c.ClusterCipherSuites
if c2.ClusterCipherSuites != "" {
result.ClusterCipherSuites = c2.ClusterCipherSuites
}
2017-02-24 10:45:29 -05:00
result.EnableUI = c.EnableUI
if c2.EnableUI {
result.EnableUI = c2.EnableUI
}
result.EnableRawEndpoint = c.EnableRawEndpoint
if c2.EnableRawEndpoint {
result.EnableRawEndpoint = c2.EnableRawEndpoint
}
result.EnableIntrospectionEndpoint = c.EnableIntrospectionEndpoint
if c2.EnableIntrospectionEndpoint {
result.EnableIntrospectionEndpoint = c2.EnableIntrospectionEndpoint
}
result.APIAddr = c.APIAddr
if c2.APIAddr != "" {
result.APIAddr = c2.APIAddr
}
result.ClusterAddr = c.ClusterAddr
if c2.ClusterAddr != "" {
result.ClusterAddr = c2.ClusterAddr
}
// Retain raw value so that it can be assigned to storage objects
result.DisableClustering = c.DisableClustering
result.DisableClusteringRaw = c.DisableClusteringRaw
if c2.DisableClusteringRaw != nil {
result.DisableClustering = c2.DisableClustering
result.DisableClusteringRaw = c2.DisableClusteringRaw
}
2017-04-13 14:22:53 -04:00
result.PluginDirectory = c.PluginDirectory
if c2.PluginDirectory != "" {
result.PluginDirectory = c2.PluginDirectory
}
result.PluginTmpdir = c.PluginTmpdir
if c2.PluginTmpdir != "" {
result.PluginTmpdir = c2.PluginTmpdir
}
result.PluginFileUid = c.PluginFileUid
if c2.PluginFileUid != 0 {
result.PluginFileUid = c2.PluginFileUid
}
result.PluginFilePermissions = c.PluginFilePermissions
if c2.PluginFilePermissionsRaw != nil {
result.PluginFilePermissions = c2.PluginFilePermissions
result.PluginFilePermissionsRaw = c2.PluginFilePermissionsRaw
}
2018-08-24 12:09:03 -04:00
result.DisablePerformanceStandby = c.DisablePerformanceStandby
if c2.DisablePerformanceStandby {
result.DisablePerformanceStandby = c2.DisablePerformanceStandby
}
result.AllowAuditLogPrefixing = c.AllowAuditLogPrefixing
if c2.AllowAuditLogPrefixing {
result.AllowAuditLogPrefixing = c2.AllowAuditLogPrefixing
}
2018-05-30 08:34:27 -04:00
result.DisableSealWrap = c.DisableSealWrap
if c2.DisableSealWrap {
result.DisableSealWrap = c2.DisableSealWrap
}
2018-10-23 15:03:17 -04:00
result.DisableIndexing = c.DisableIndexing
if c2.DisableIndexing {
result.DisableIndexing = c2.DisableIndexing
}
result.EnableResponseHeaderHostname = c.EnableResponseHeaderHostname
if c2.EnableResponseHeaderHostname {
result.EnableResponseHeaderHostname = c2.EnableResponseHeaderHostname
}
result.LogRequestsLevel = c.LogRequestsLevel
if c2.LogRequestsLevel != "" {
result.LogRequestsLevel = c2.LogRequestsLevel
}
result.DetectDeadlocks = c.DetectDeadlocks
if c2.DetectDeadlocks != "" {
result.DetectDeadlocks = c2.DetectDeadlocks
}
result.DisablePrintableCheck = c.DisablePrintableCheck
if c2.DisablePrintableCheckRaw != nil {
result.DisablePrintableCheck = c2.DisablePrintableCheck
}
result.Observations = c.Observations
if c2.Observations != nil {
if result.Observations == nil {
result.Observations = &observations.ObservationSystemConfig{}
}
if c2.Observations.LedgerPath != "" {
result.Observations.LedgerPath = c2.Observations.LedgerPath
}
result.Observations.TypePrefixDenylist = append(result.Observations.TypePrefixDenylist, c2.Observations.TypePrefixDenylist...)
result.Observations.TypePrefixAllowlist = append(result.Observations.TypePrefixAllowlist, c2.Observations.TypePrefixAllowlist...)
if c2.Observations.FileMode != "" {
result.Observations.FileMode = c2.Observations.FileMode
}
}
result.ImpreciseLeaseRoleTracking = c.ImpreciseLeaseRoleTracking
if c2.ImpreciseLeaseRoleTracking {
result.ImpreciseLeaseRoleTracking = c2.ImpreciseLeaseRoleTracking
}
result.EnableResponseHeaderRaftNodeID = c.EnableResponseHeaderRaftNodeID
if c2.EnableResponseHeaderRaftNodeID {
result.EnableResponseHeaderRaftNodeID = c2.EnableResponseHeaderRaftNodeID
}
result.LicensePath = c.LicensePath
if c2.LicensePath != "" {
result.LicensePath = c2.LicensePath
}
result.EnablePostUnsealTrace = c.EnablePostUnsealTrace
if c2.EnablePostUnsealTrace {
result.EnablePostUnsealTrace = c2.EnablePostUnsealTrace
}
result.PostUnsealTraceDir = c.PostUnsealTraceDir
if c2.PostUnsealTraceDir != "" {
result.PostUnsealTraceDir = c2.PostUnsealTraceDir
}
result.ReportingScanDirectory = c.ReportingScanDirectory
if c2.ReportingScanDirectory != "" {
result.ReportingScanDirectory = c2.ReportingScanDirectory
}
// Use values from top-level configuration for storage if set
if storage := result.Storage; storage != nil {
if result.APIAddr != "" {
storage.RedirectAddr = result.APIAddr
}
if result.ClusterAddr != "" {
storage.ClusterAddr = result.ClusterAddr
}
if result.DisableClusteringRaw != nil {
storage.DisableClustering = result.DisableClustering
}
}
if haStorage := result.HAStorage; haStorage != nil {
if result.APIAddr != "" {
haStorage.RedirectAddr = result.APIAddr
}
if result.ClusterAddr != "" {
haStorage.ClusterAddr = result.ClusterAddr
}
if result.DisableClusteringRaw != nil {
haStorage.DisableClustering = result.DisableClustering
}
}
2023-07-06 03:51:43 -04:00
result.AdministrativeNamespacePath = c.AdministrativeNamespacePath
if c2.AdministrativeNamespacePath != "" {
result.AdministrativeNamespacePath = c2.AdministrativeNamespacePath
}
result.entConfig = c.entConfig.Merge(c2.entConfig)
result.Experiments = mergeExperiments(c.Experiments, c2.Experiments)
result.EnableUnauthenticatedAccess = c.EnableUnauthenticatedAccess
if len(c2.EnableUnauthenticatedAccess) > 0 {
result.EnableUnauthenticatedAccess = c2.EnableUnauthenticatedAccess
}
2015-03-12 18:21:11 -04:00
return result
}
2015-03-12 18:30:07 -04:00
// LoadConfig loads the configuration at the given path, regardless if
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
// it's a file or directory.
func LoadConfig(path string) (*Config, error) {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
cfg, _, err := LoadConfigCheckDuplicate(path)
return cfg, err
}
// LoadConfigCheckDuplicate is the same as the above function but also checks for duplicate attributes
// TODO (HCL_DUP_KEYS_DEPRECATION): keep only LoadConfig once deprecation is complete
func LoadConfigCheckDuplicate(path string) (cfg *Config, duplicate bool, err error) {
2015-03-12 18:30:07 -04:00
fi, err := os.Stat(path)
if err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, false, err
2015-03-12 18:30:07 -04:00
}
if fi.IsDir() {
// check permissions on the config directory
var enableFilePermissionsCheck bool
if enableFilePermissionsCheckEnv := os.Getenv(consts.VaultEnableFilePermissionsCheckEnv); enableFilePermissionsCheckEnv != "" {
var err error
enableFilePermissionsCheck, err = strconv.ParseBool(enableFilePermissionsCheckEnv)
if err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, false, errors.New("Error parsing the environment variable VAULT_ENABLE_FILE_PERMISSIONS_CHECK")
}
}
f, err := os.Open(path)
if err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, false, err
}
defer f.Close()
if enableFilePermissionsCheck {
err = osutil.OwnerPermissionsMatchFile(f, 0, 0)
if err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, false, err
}
}
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
cfg, duplicate, err = LoadConfigDirCheckDuplicate(path)
if err != nil {
return nil, duplicate, err
}
} else {
cfg, duplicate, err = LoadConfigFileCheckDuplicate(path)
if err != nil {
return nil, duplicate, err
}
}
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
cfg, err = CheckConfig(cfg)
return cfg, duplicate, err
}
func CheckConfig(c *Config) (*Config, error) {
if err := c.checkSealConfig(); err != nil {
return nil, err
}
sealMap := make(map[string]*configutil.KMS)
for _, seal := range c.Seals {
Add vault community changes for the Seal HA project. (#22515) * Seal HA: Use new SealWrappedValue type to abstract seal wrapped values Introduce SealWrappedValue to abstract seal wrapped values. Make SealWrappedValue capable of marshalling into a BlobInfo, when there is plaintext or a single encryption, or to a custom serialization consisting of a header, length and a marshalled MultiWrapValue protobuf. * Vault-13769: Support configuring and using multiple seals for unsealing * Make sealWrapBackend start using multiple seals * Make seal.Access no longer implement wrapping.Wrapper. Instead, add the Encrypt and Decrypt methods to the Access interface. * Make raft snapshot system use funcs SealWrapValue + UnsealWrapValue. Move the snapshot.Sealer implementation to the vault package to avoid circular imports. * Update sealWrapBackend to use multiple seals for encryption. Use all the encryption wrappers when storing seal wrapped values. Try do decrypt using the highest priority wrapper, but try all combinations of encrypted values and wrappers if necessary. * Allow the use of multiple seals for entropy augmentation Add seal_name variable in entropy stanza Add new MultiSourcer to accommodate the new entropy augmentation behavior. * Individually health check each wrapper, and add a sys/seal-backend-status endpoint. * Address a race, and also a failed test mock that I didn't catch * Track partial wrapping failures... ... where one or more but not all access.Encrypts fail for a given write. Note these failures by adding a time ordered UUID storage entry containing the path in a special subdirectory of root storage. Adds a callback pattern to accomplish this, with certain high value writes like initial barrier key storage not allowing a partial failure. The followup work would be to detect return to health and iterate through these storage entries, rewrapping. * Add new data structure to track seal config generation (#4492) * Add new data structure to track seal config generation * Remove import cycle * Fix undefined variable errors * update comment * Update setSeal response * Fix setSealResponse in operator_diagnose * Scope the wrapper health check locks individually (#4491) * Refactor setSeal function in server.go. (#4505) Refactor setSeal function in server.go. * Decouple CreateSecureRandomReaderFunc from seal package. Instead of using a list of seal.SealInfo structs, make CreateSecureRandomReaderFunc use a list of new EntropySourcerInfo structs. This brakes the denpency of package configutil on the seal package. * Move SealGenerationInfo tracking to the seal Access. * Move SealGenerationInfo tracking to the seal Access. The SealGenerationInfo is now kept track by a Seal's Access instead of by the Config object. The access implementation now records the correct generation number on seal wrapped values. * Only store and read SealGenerationInfo if VAULT_ENABLE_SEAL_HA_BETA is true. * Add MultiWrapValue protobuf message MultiWrapValue can be used to keep track of different encryptions of a value. --------- Co-authored-by: Victor Rodriguez <vrizo@hashicorp.com> * Use generation to determine if a seal wrapped value is up-to-date. (#4542) * Add logging to seal Access implementation. * Seal HA buf format run (#4561) * Run buf format. * Add buf.lock to ensure go-kms-wrapping module is imported. * Vault-18958: Add unit tests for config checks * Add safety logic for seal configuration changes * Revert "Add safety logic for seal configuration changes" This reverts commit 7fec48035a5cf274e5a4d98901716d08d766ce90. * changes and tests for checking seal config * add ent tests * remove check for empty name and add type into test cases * add error message for empty name * fix no seals test --------- Co-authored-by: divyapola5 <divya@hashicorp.com> * Handle migrations between single-wrapper and multi-wrapper autoSeals * Extract method SetPhysicalSealConfig. * Extract function physicalSealConfig. The extracted function is the only code now reading SealConfig entries from storage. * Extract function setPhysicalSealConfig. The extracted function is the only code now writing SealConfig entries from storage (except for migration from the old recovery config path). * Move SealConfig to new file vault/seal_config.go. * Add SealConfigType quasy-enumeration. SealConfigType is to serve as the typed values for field SealConfig.Type. * Rename Seal.RecoveryType to RecoverySealConfigType. Make RecoverySealConfigType return a SealConfigType instead of a string. * Rename Seal.BarrierType to BarrierSealConfigType. Make BarrierSealConfigType return a SealConfigType. Remove seal.SealType (really a two-step rename to SealConfigType). * Add Seal methods ClearBarrierConfig and ClearRecoveryConfig. * Handle autoseal <-> multiseal migrations. While going between single-wrapper and multiple-wrapper autoseals are not migrations that require an unwrap seal (such as going from shamir to autoseal), the stored "barrier" SealConfig needs to be updated in these cases. Specifically, the value of SealConfg.Type is "multiseal" for autoSeals that have more than one wrapper; on the other hand, for autoseals with a single wrapper, SealConfig.Type is the type of the wrapper. * Remove error return value from NewAutoSeal constructor. * Automatically rewrap partially seal wrapped values on an interval * Add in rewrapping of partially wrapped values on an interval, regardless of seal health/status. * Don't set SealGenerationInfo Rewrapped flag in the partial rewrap call. * Unexport the SealGenerationInfo's Rewrapped field, add a mutex to it for thread safe access, and add accessor methods for it. * Add a success callback to the manual seal rewrap process that updates the SealGenerationInfo's rewrapped field. This is done via a callback to avoid an import cycle in the SealRewrap code. * Fix a failing seal wrap backend test which was broken by the unexporting of SealGenerationInfo's Rewrapped field. * Nil check the seal rewrap success callback before calling it. * Change SealGenerationInfo rewrapped parameter to an atomic.Bool rather than a sync.RWMutex for simplicity and performance. * Add nil check for SealAccess before updating SealGenerationInfo rewrapped status during seal rewrap call. * Update partial rewrap check interval from 10 seconds to 1 minute. * Update a reference to SealGenerationInfo Rewrapped field to use new getter method. * Fix up some data raciness in partial rewrapping. * Account for possibly nil storage entry when retrieving partially wrapped value. * Allow multi-wrapper autoSeals to include disabled seal wrappers. * Restore propagation of wrapper configuration errors by setSeal. Function setSeal is meant to propagate non KeyNotFound errors returned by calls to configutil.ConfigureWrapper. * Remove unused Access methods SetConfig and Type. * Allow multi-wrapper autoSeals to include disabled seal wrappers. Make it possible for an autoSeal that uses multiple wrappers to include disabled wrappers that can be used to decrypt entries, but are skipped for encryption. e an unwrapSeal when there are disabled seals. * Fix bug with not providing name (#4580) * add suffix to name defaults * add comment * only change name for disabled seal * Only attempt to rewrap partial values when all seals are healthy. * Only attempt to rewrap partial values when all seals are healthy. * Change logging level from info to debug for notice about rewrap skipping based on seal health. * Remove stale TODOs and commented out code. --------- Co-authored-by: rculpepper <rculpepper@hashicorp.com> Co-authored-by: Larroyo <95649169+DeLuci@users.noreply.github.com> Co-authored-by: Scott G. Miller <smiller@hashicorp.com> Co-authored-by: Divya Pola <87338962+divyapola5@users.noreply.github.com> Co-authored-by: Matt Schultz <matt.schultz@hashicorp.com> Co-authored-by: divyapola5 <divya@hashicorp.com> Co-authored-by: Rachel Culpepper <84159930+rculpepper@users.noreply.github.com>
2023-08-24 11:40:01 -04:00
if seal.Name == "" {
return nil, errors.New("seals: seal name is empty")
}
if _, ok := sealMap[seal.Name]; ok {
return nil, errors.New("seals: seal names must be unique")
}
sealMap[seal.Name] = seal
}
return c, nil
2015-03-12 18:30:07 -04:00
}
2015-03-12 18:21:11 -04:00
// LoadConfigFile loads the configuration from the given file.
func LoadConfigFile(path string) (*Config, error) {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
cfg, _, err := LoadConfigFileCheckDuplicate(path)
return cfg, err
}
// LoadConfigFileCheckDuplicate is the same as the above function but also checks for duplicate attributes
// TODO (HCL_DUP_KEYS_DEPRECATION): keep only ParseConfig once deprecation is complete
func LoadConfigFileCheckDuplicate(path string) (cfg *Config, duplicate bool, err error) {
// Open the file
f, err := os.Open(path)
if err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, false, err
}
defer f.Close()
2015-03-12 18:21:11 -04:00
// Read the file
d, err := io.ReadAll(f)
2015-03-12 18:21:11 -04:00
if err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, false, err
2015-03-12 18:21:11 -04:00
}
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
conf, duplicate, err := ParseConfigCheckDuplicate(string(d), path)
if err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
}
var enableFilePermissionsCheck bool
if enableFilePermissionsCheckEnv := os.Getenv(consts.VaultEnableFilePermissionsCheckEnv); enableFilePermissionsCheckEnv != "" {
var err error
enableFilePermissionsCheck, err = strconv.ParseBool(enableFilePermissionsCheckEnv)
if err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, errors.New("Error parsing the environment variable VAULT_ENABLE_FILE_PERMISSIONS_CHECK")
}
}
if enableFilePermissionsCheck {
// check permissions of the config file
err = osutil.OwnerPermissionsMatchFile(f, 0, 0)
if err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
}
// check permissions of the plugin directory
if conf.PluginDirectory != "" {
err = osutil.OwnerPermissionsMatch(conf.PluginDirectory, conf.PluginFileUid, conf.PluginFilePermissions)
if err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
}
}
}
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return conf, duplicate, nil
}
2015-03-12 18:21:11 -04:00
func ParseConfig(d, source string) (*Config, error) {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
cfg, _, err := ParseConfigCheckDuplicate(d, source)
return cfg, err
}
// TODO (HCL_DUP_KEYS_DEPRECATION): keep only ParseConfig once deprecation is complete
func ParseConfigCheckDuplicate(d, source string) (cfg *Config, duplicate bool, err error) {
2015-03-12 18:21:11 -04:00
// Parse!
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
obj, duplicate, err := random.ParseAndCheckForDuplicateHclAttributes(d)
2015-03-12 18:21:11 -04:00
if err != nil {
if strings.Contains(err.Error(), "was already set. Each argument can only be defined once") {
knownPossibleAttributeDupErrors := []string{"retry_join", "transform", "listener"}
for _, s := range knownPossibleAttributeDupErrors {
if strings.Contains(err.Error(), fmt.Sprintf("The argument %q at", s)) {
return nil, duplicate, fmt.Errorf("%w (if using the attribute syntax %s = [...], change it to the block syntax %s { ... })", err, s, s)
}
}
}
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
2015-03-12 18:21:11 -04:00
}
// Start building the result
result := NewConfig()
if err := hcl.DecodeObject(result, obj); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
}
if rendered, err := configutil.ParseSingleIPTemplate(result.APIAddr); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
} else {
result.APIAddr = rendered
}
if rendered, err := configutil.ParseSingleIPTemplate(result.ClusterAddr); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
} else {
result.ClusterAddr = rendered
}
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
sharedConfig, dup, err := configutil.ParseConfigCheckDuplicate(d)
if err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
}
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
duplicate = duplicate || dup
result.SharedConfig = sharedConfig
if result.MaxLeaseTTLRaw != nil {
if result.MaxLeaseTTL, err = parseutil.ParseDurationSecond(result.MaxLeaseTTLRaw); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
2015-07-30 13:21:41 -04:00
}
}
if result.DefaultLeaseTTLRaw != nil {
if result.DefaultLeaseTTL, err = parseutil.ParseDurationSecond(result.DefaultLeaseTTLRaw); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
}
}
if result.RemoveIrrevocableLeaseAfterRaw != nil {
if result.RemoveIrrevocableLeaseAfter, err = parseutil.ParseDurationSecond(result.RemoveIrrevocableLeaseAfterRaw); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
}
}
if result.EnableUIRaw != nil {
if result.EnableUI, err = parseutil.ParseBool(result.EnableUIRaw); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
}
}
if result.DisableCacheRaw != nil {
if result.DisableCache, err = parseutil.ParseBool(result.DisableCacheRaw); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
}
}
if result.DisablePrintableCheckRaw != nil {
if result.DisablePrintableCheck, err = parseutil.ParseBool(result.DisablePrintableCheckRaw); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
}
}
if result.EnableRawEndpointRaw != nil {
if result.EnableRawEndpoint, err = parseutil.ParseBool(result.EnableRawEndpointRaw); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
}
}
if result.EnableIntrospectionEndpointRaw != nil {
if result.EnableIntrospectionEndpoint, err = parseutil.ParseBool(result.EnableIntrospectionEndpointRaw); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
}
}
if result.DisableClusteringRaw != nil {
if result.DisableClustering, err = parseutil.ParseBool(result.DisableClusteringRaw); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
}
}
if result.PluginFilePermissionsRaw != nil {
octalPermissionsString, err := parseutil.ParseString(result.PluginFilePermissionsRaw)
if err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
}
pluginFilePermissions, err := strconv.ParseInt(octalPermissionsString, 8, 64)
if err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
}
if pluginFilePermissions < math.MinInt || pluginFilePermissions > math.MaxInt {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, fmt.Errorf("file permission value %v cannot be safely cast to int: exceeds bounds (%v, %v)", pluginFilePermissions, math.MinInt, math.MaxInt)
}
result.PluginFilePermissions = int(pluginFilePermissions)
}
if result.DisableSentinelTraceRaw != nil {
if result.DisableSentinelTrace, err = parseutil.ParseBool(result.DisableSentinelTraceRaw); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
}
}
2018-08-24 12:09:03 -04:00
if result.DisablePerformanceStandbyRaw != nil {
if result.DisablePerformanceStandby, err = parseutil.ParseBool(result.DisablePerformanceStandbyRaw); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
2018-08-24 12:09:03 -04:00
}
}
if result.AllowAuditLogPrefixingRaw != nil {
if result.AllowAuditLogPrefixing, err = parseutil.ParseBool(result.AllowAuditLogPrefixingRaw); err != nil {
return nil, duplicate, err
}
}
2018-05-30 08:34:27 -04:00
if result.DisableSealWrapRaw != nil {
if result.DisableSealWrap, err = parseutil.ParseBool(result.DisableSealWrapRaw); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
2018-05-30 08:34:27 -04:00
}
}
2018-10-23 15:09:35 -04:00
if result.DisableIndexingRaw != nil {
if result.DisableIndexing, err = parseutil.ParseBool(result.DisableIndexingRaw); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
2018-10-23 15:03:17 -04:00
}
}
if result.EnableResponseHeaderHostnameRaw != nil {
if result.EnableResponseHeaderHostname, err = parseutil.ParseBool(result.EnableResponseHeaderHostnameRaw); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
}
}
if result.LogRequestsLevelRaw != nil {
result.LogRequestsLevel = strings.ToLower(strings.TrimSpace(result.LogRequestsLevelRaw.(string)))
result.LogRequestsLevelRaw = ""
}
if result.EnableResponseHeaderRaftNodeIDRaw != nil {
if result.EnableResponseHeaderRaftNodeID, err = parseutil.ParseBool(result.EnableResponseHeaderRaftNodeIDRaw); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, err
}
}
list, ok := obj.Node.(*ast.ObjectList)
if !ok {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, fmt.Errorf("error parsing: file doesn't contain a root object")
2015-03-12 18:21:11 -04:00
}
// Look for storage but still support old backend
if o := list.Filter("storage"); len(o.Items) > 0 {
delete(result.UnusedKeys, "storage")
if err := ParseStorage(result, o, "storage"); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, fmt.Errorf("error parsing 'storage': %w", err)
}
result.found(result.Storage.Type, result.Storage.Type)
} else {
delete(result.UnusedKeys, "backend")
if o := list.Filter("backend"); len(o.Items) > 0 {
if err := ParseStorage(result, o, "backend"); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, fmt.Errorf("error parsing 'backend': %w", err)
}
}
}
2015-07-14 18:27:18 -04:00
if o := list.Filter("ha_storage"); len(o.Items) > 0 {
delete(result.UnusedKeys, "ha_storage")
if err := parseHAStorage(result, o, "ha_storage"); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, fmt.Errorf("error parsing 'ha_storage': %w", err)
}
} else {
if o := list.Filter("ha_backend"); len(o.Items) > 0 {
delete(result.UnusedKeys, "ha_backend")
if err := parseHAStorage(result, o, "ha_backend"); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, fmt.Errorf("error parsing 'ha_backend': %w", err)
}
2015-07-14 18:27:18 -04:00
}
}
// Parse service discovery
if o := list.Filter("service_registration"); len(o.Items) > 0 {
delete(result.UnusedKeys, "service_registration")
if err := parseServiceRegistration(result, o, "service_registration"); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, fmt.Errorf("error parsing 'service_registration': %w", err)
}
}
if err := validateExperiments(result.Experiments); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, fmt.Errorf("error validating experiment(s) from config: %w", err)
}
if err := result.parseConfig(list, source); err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, fmt.Errorf("error parsing enterprise config: %w", err)
2020-02-14 19:39:13 -05:00
}
// Remove all unused keys from Config that were satisfied by SharedConfig.
result.UnusedKeys = configutil.UnusedFieldDifference(result.UnusedKeys, nil, append(result.FoundKeys, sharedConfig.FoundKeys...))
// Assign file info
for _, v := range result.UnusedKeys {
for i := range v {
v[i].Filename = source
}
}
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return result, duplicate, nil
2015-07-14 18:27:18 -04:00
}
func ExperimentsFromEnvAndCLI(config *Config, envKey string, flagExperiments []string) error {
if envExperimentsRaw := os.Getenv(envKey); envExperimentsRaw != "" {
envExperiments := strings.Split(envExperimentsRaw, ",")
err := validateExperiments(envExperiments)
if err != nil {
return fmt.Errorf("error validating experiment(s) from environment variable %q: %w", envKey, err)
}
config.Experiments = mergeExperiments(config.Experiments, envExperiments)
}
if len(flagExperiments) != 0 {
err := validateExperiments(flagExperiments)
if err != nil {
return fmt.Errorf("error validating experiment(s) from command line flag: %w", err)
}
config.Experiments = mergeExperiments(config.Experiments, flagExperiments)
}
return nil
}
// validateExperiments checks each experiment is a known experiment.
func validateExperiments(experiments []string) error {
var invalid []string
for _, experiment := range experiments {
if !strutil.StrListContains(validExperiments, experiment) {
invalid = append(invalid, experiment)
}
}
if len(invalid) != 0 {
return fmt.Errorf("valid experiment(s) are %s, but received the following invalid experiment(s): %s",
strings.Join(validExperiments, ", "),
strings.Join(invalid, ", "))
}
return nil
}
// mergeExperiments returns the logical OR of the two sets.
func mergeExperiments(left, right []string) []string {
processed := map[string]struct{}{}
var result []string
for _, l := range left {
if _, seen := processed[l]; !seen {
result = append(result, l)
}
processed[l] = struct{}{}
}
for _, r := range right {
if _, seen := processed[r]; !seen {
result = append(result, r)
processed[r] = struct{}{}
}
}
return result
}
2015-03-12 18:21:11 -04:00
// LoadConfigDir loads all the configurations in the given directory
// in alphabetical order.
func LoadConfigDir(dir string) (*Config, error) {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
cfg, _, err := LoadConfigDirCheckDuplicate(dir)
return cfg, err
}
// LoadConfigDirCheckDuplicate is the same as the above but checks for duplciate HCL attributes
// TODO (HCL_DUP_KEYS_DEPRECATION): keep only LoadConfigDir once deprecation is complete
func LoadConfigDirCheckDuplicate(dir string) (cfg *Config, duplicate bool, err error) {
2015-03-12 18:21:11 -04:00
f, err := os.Open(dir)
if err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, false, err
2015-03-12 18:21:11 -04:00
}
defer f.Close()
fi, err := f.Stat()
if err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, false, err
2015-03-12 18:21:11 -04:00
}
if !fi.IsDir() {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, false, fmt.Errorf("configuration path must be a directory: %q", dir)
2015-03-12 18:21:11 -04:00
}
var files []string
err = nil
for err != io.EOF {
var fis []os.FileInfo
fis, err = f.Readdir(128)
if err != nil && err != io.EOF {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, false, err
2015-03-12 18:21:11 -04:00
}
for _, fi := range fis {
// Ignore directories
if fi.IsDir() {
continue
}
// Only care about files that are valid to load.
name := fi.Name()
skip := true
if strings.HasSuffix(name, ".hcl") {
skip = false
} else if strings.HasSuffix(name, ".json") {
skip = false
}
if skip || isTemporaryFile(name) {
continue
}
path := filepath.Join(dir, name)
files = append(files, path)
}
}
result := NewConfig()
2015-03-12 18:21:11 -04:00
for _, f := range files {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
config, dup, err := LoadConfigFileCheckDuplicate(f)
2015-03-12 18:21:11 -04:00
if err != nil {
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return nil, duplicate, fmt.Errorf("error loading %q: %w", f, err)
2015-03-12 18:21:11 -04:00
}
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
duplicate = duplicate || dup
2015-03-12 18:21:11 -04:00
if result == nil {
result = config
} else {
result = result.Merge(config)
}
}
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386) * upgrade hcl dependency on api pkg This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier. * upgrade hcl dependency in vault and sdk pkgs * upgrade hcl dependency in vault and sdk pkgs * add CLI warnings to commands that take a config - vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server * ignore duplicates on ParseKMSes function * Extend policy parsing functions and warn on policy store * Add warning on policy fmt with duplicate attributes * Add warnings when creating/updating policy with duplicate HCL attrs * Add log warning when switchedGetPolicy finds duplicate attrs Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal * Print log warnings when token inline policy has duplicate attrs No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen) * add changelog and deprecation notice * add missing copywrite notice * fix copy-paste mistake good thing it was covered by unit tests * Fix manual parsing of telemetry field in SharedConfig This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: https://github.com/hashicorp/hcl/commit/e80118accb521e47bc5b93104bf46c67d89d2242 This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates * Fix linter complaints * Update command/base_predict.go Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> * address review * remove copywrite headers * re-add copywrite headers * make fmt * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com> * undo changes to deprecation.mdx * remove deprecation doc * fix conflict with changes from main --------- Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 15:02:07 -04:00
return result, duplicate, nil
2015-03-12 18:21:11 -04:00
}
// isTemporaryFile returns true or false depending on whether the
// provided file name is a temporary file for the following editors:
// emacs or vim.
func isTemporaryFile(name string) bool {
return strings.HasSuffix(name, "~") || // vim
strings.HasPrefix(name, ".#") || // emacs
(strings.HasPrefix(name, "#") && strings.HasSuffix(name, "#")) // emacs
}
func ParseStorage(result *Config, list *ast.ObjectList, name string) error {
if len(list.Items) > 1 {
return fmt.Errorf("only one %q block is permitted", name)
}
// Get our item
item := list.Items[0]
key := name
if len(item.Keys) > 0 {
key = item.Keys[0].Token.Value().(string)
}
var config map[string]interface{}
if err := hcl.DecodeObject(&config, item.Val); err != nil {
return multierror.Prefix(err, fmt.Sprintf("%s.%s:", name, key))
}
m := make(map[string]string)
VAULT-33008: ipv6: always display RFC-5952 §4 conformant addresses (#29228) USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for IPv6-Only capabilities. This section requires that whenever Vault displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that _all_ IPv6 addresses must conform to RFC-5952 §4 text representation recommendations[2]. These recommendations do not prevent us from accepting RFC-4241[3] IPv6 addresses, however, whenever these same addresses are displayed they must conform to the strict RFC-5952 §4 guidelines. This PR implements handling of IPv6 address conformance in our `vault server` routine. We handle conformance normalization for all server, http_proxy, listener, seal, storage and telemetry configuration where an input could contain an IPv6 address, whether configured via an HCL file or via corresponding environment variables. The approach I've taken is to handle conformance normalization at parse time to ensure that all log output and subsequent usage inside of Vaults various subsystems always reference a conformant address, that way we don't need concern ourselves with conformance later. This approach ought to be backwards compatible to prior loose address configuration requirements, with the understanding that going forward all IPv6 representation will be strict regardless of what has been configured. In many cases I've updated our various parser functions to call the new `configutil.NormalizeAddr()` to apply conformance normalization. Others required no changes because they rely on standard library URL string output, which always displays IPv6 URLs in a conformant way. Not included in this changes is any other vault exec mode other than server. Client, operator commands, agent mode, proxy mode, etc. will be included in subsequent changes if necessary. [0]: https://www.nist.gov/publications/usgv6-profile [1]: https://www.nist.gov/publications/nist-ipv6-profile [2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4 [3]: https://www.rfc-editor.org/rfc/rfc4291 Signed-off-by: Ryan Cragun <me@ryan.ec>
2025-01-27 16:14:28 -05:00
for k, v := range config {
vStr, ok := v.(string)
if ok {
VAULT-33008: ipv6: always display RFC-5952 §4 conformant addresses (#29228) USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for IPv6-Only capabilities. This section requires that whenever Vault displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that _all_ IPv6 addresses must conform to RFC-5952 §4 text representation recommendations[2]. These recommendations do not prevent us from accepting RFC-4241[3] IPv6 addresses, however, whenever these same addresses are displayed they must conform to the strict RFC-5952 §4 guidelines. This PR implements handling of IPv6 address conformance in our `vault server` routine. We handle conformance normalization for all server, http_proxy, listener, seal, storage and telemetry configuration where an input could contain an IPv6 address, whether configured via an HCL file or via corresponding environment variables. The approach I've taken is to handle conformance normalization at parse time to ensure that all log output and subsequent usage inside of Vaults various subsystems always reference a conformant address, that way we don't need concern ourselves with conformance later. This approach ought to be backwards compatible to prior loose address configuration requirements, with the understanding that going forward all IPv6 representation will be strict regardless of what has been configured. In many cases I've updated our various parser functions to call the new `configutil.NormalizeAddr()` to apply conformance normalization. Others required no changes because they rely on standard library URL string output, which always displays IPv6 URLs in a conformant way. Not included in this changes is any other vault exec mode other than server. Client, operator commands, agent mode, proxy mode, etc. will be included in subsequent changes if necessary. [0]: https://www.nist.gov/publications/usgv6-profile [1]: https://www.nist.gov/publications/nist-ipv6-profile [2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4 [3]: https://www.rfc-editor.org/rfc/rfc4291 Signed-off-by: Ryan Cragun <me@ryan.ec>
2025-01-27 16:14:28 -05:00
var err error
m[k], err = normalizeStorageConfigAddresses(key, k, vStr)
if err != nil {
return err
}
continue
}
VAULT-33008: ipv6: always display RFC-5952 §4 conformant addresses (#29228) USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for IPv6-Only capabilities. This section requires that whenever Vault displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that _all_ IPv6 addresses must conform to RFC-5952 §4 text representation recommendations[2]. These recommendations do not prevent us from accepting RFC-4241[3] IPv6 addresses, however, whenever these same addresses are displayed they must conform to the strict RFC-5952 §4 guidelines. This PR implements handling of IPv6 address conformance in our `vault server` routine. We handle conformance normalization for all server, http_proxy, listener, seal, storage and telemetry configuration where an input could contain an IPv6 address, whether configured via an HCL file or via corresponding environment variables. The approach I've taken is to handle conformance normalization at parse time to ensure that all log output and subsequent usage inside of Vaults various subsystems always reference a conformant address, that way we don't need concern ourselves with conformance later. This approach ought to be backwards compatible to prior loose address configuration requirements, with the understanding that going forward all IPv6 representation will be strict regardless of what has been configured. In many cases I've updated our various parser functions to call the new `configutil.NormalizeAddr()` to apply conformance normalization. Others required no changes because they rely on standard library URL string output, which always displays IPv6 URLs in a conformant way. Not included in this changes is any other vault exec mode other than server. Client, operator commands, agent mode, proxy mode, etc. will be included in subsequent changes if necessary. [0]: https://www.nist.gov/publications/usgv6-profile [1]: https://www.nist.gov/publications/nist-ipv6-profile [2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4 [3]: https://www.rfc-editor.org/rfc/rfc4291 Signed-off-by: Ryan Cragun <me@ryan.ec>
2025-01-27 16:14:28 -05:00
var err error
var vBytes []byte
// Raft's retry_join requires special normalization due to its complexity
if key == "raft" && k == "retry_join" {
vBytes, err = normalizeRaftRetryJoin(v)
if err != nil {
return err
}
} else {
vBytes, err = json.Marshal(v)
if err != nil {
return err
}
}
VAULT-33008: ipv6: always display RFC-5952 §4 conformant addresses (#29228) USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for IPv6-Only capabilities. This section requires that whenever Vault displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that _all_ IPv6 addresses must conform to RFC-5952 §4 text representation recommendations[2]. These recommendations do not prevent us from accepting RFC-4241[3] IPv6 addresses, however, whenever these same addresses are displayed they must conform to the strict RFC-5952 §4 guidelines. This PR implements handling of IPv6 address conformance in our `vault server` routine. We handle conformance normalization for all server, http_proxy, listener, seal, storage and telemetry configuration where an input could contain an IPv6 address, whether configured via an HCL file or via corresponding environment variables. The approach I've taken is to handle conformance normalization at parse time to ensure that all log output and subsequent usage inside of Vaults various subsystems always reference a conformant address, that way we don't need concern ourselves with conformance later. This approach ought to be backwards compatible to prior loose address configuration requirements, with the understanding that going forward all IPv6 representation will be strict regardless of what has been configured. In many cases I've updated our various parser functions to call the new `configutil.NormalizeAddr()` to apply conformance normalization. Others required no changes because they rely on standard library URL string output, which always displays IPv6 URLs in a conformant way. Not included in this changes is any other vault exec mode other than server. Client, operator commands, agent mode, proxy mode, etc. will be included in subsequent changes if necessary. [0]: https://www.nist.gov/publications/usgv6-profile [1]: https://www.nist.gov/publications/nist-ipv6-profile [2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4 [3]: https://www.rfc-editor.org/rfc/rfc4291 Signed-off-by: Ryan Cragun <me@ryan.ec>
2025-01-27 16:14:28 -05:00
m[k] = string(vBytes)
}
// Pull out the redirect address since it's common to all backends
var redirectAddr string
if v, ok := m["redirect_addr"]; ok {
VAULT-33008: ipv6: always display RFC-5952 §4 conformant addresses (#29228) USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for IPv6-Only capabilities. This section requires that whenever Vault displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that _all_ IPv6 addresses must conform to RFC-5952 §4 text representation recommendations[2]. These recommendations do not prevent us from accepting RFC-4241[3] IPv6 addresses, however, whenever these same addresses are displayed they must conform to the strict RFC-5952 §4 guidelines. This PR implements handling of IPv6 address conformance in our `vault server` routine. We handle conformance normalization for all server, http_proxy, listener, seal, storage and telemetry configuration where an input could contain an IPv6 address, whether configured via an HCL file or via corresponding environment variables. The approach I've taken is to handle conformance normalization at parse time to ensure that all log output and subsequent usage inside of Vaults various subsystems always reference a conformant address, that way we don't need concern ourselves with conformance later. This approach ought to be backwards compatible to prior loose address configuration requirements, with the understanding that going forward all IPv6 representation will be strict regardless of what has been configured. In many cases I've updated our various parser functions to call the new `configutil.NormalizeAddr()` to apply conformance normalization. Others required no changes because they rely on standard library URL string output, which always displays IPv6 URLs in a conformant way. Not included in this changes is any other vault exec mode other than server. Client, operator commands, agent mode, proxy mode, etc. will be included in subsequent changes if necessary. [0]: https://www.nist.gov/publications/usgv6-profile [1]: https://www.nist.gov/publications/nist-ipv6-profile [2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4 [3]: https://www.rfc-editor.org/rfc/rfc4291 Signed-off-by: Ryan Cragun <me@ryan.ec>
2025-01-27 16:14:28 -05:00
redirectAddr = configutil.NormalizeAddr(v)
delete(m, "redirect_addr")
} else if v, ok := m["advertise_addr"]; ok {
VAULT-33008: ipv6: always display RFC-5952 §4 conformant addresses (#29228) USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for IPv6-Only capabilities. This section requires that whenever Vault displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that _all_ IPv6 addresses must conform to RFC-5952 §4 text representation recommendations[2]. These recommendations do not prevent us from accepting RFC-4241[3] IPv6 addresses, however, whenever these same addresses are displayed they must conform to the strict RFC-5952 §4 guidelines. This PR implements handling of IPv6 address conformance in our `vault server` routine. We handle conformance normalization for all server, http_proxy, listener, seal, storage and telemetry configuration where an input could contain an IPv6 address, whether configured via an HCL file or via corresponding environment variables. The approach I've taken is to handle conformance normalization at parse time to ensure that all log output and subsequent usage inside of Vaults various subsystems always reference a conformant address, that way we don't need concern ourselves with conformance later. This approach ought to be backwards compatible to prior loose address configuration requirements, with the understanding that going forward all IPv6 representation will be strict regardless of what has been configured. In many cases I've updated our various parser functions to call the new `configutil.NormalizeAddr()` to apply conformance normalization. Others required no changes because they rely on standard library URL string output, which always displays IPv6 URLs in a conformant way. Not included in this changes is any other vault exec mode other than server. Client, operator commands, agent mode, proxy mode, etc. will be included in subsequent changes if necessary. [0]: https://www.nist.gov/publications/usgv6-profile [1]: https://www.nist.gov/publications/nist-ipv6-profile [2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4 [3]: https://www.rfc-editor.org/rfc/rfc4291 Signed-off-by: Ryan Cragun <me@ryan.ec>
2025-01-27 16:14:28 -05:00
redirectAddr = configutil.NormalizeAddr(v)
delete(m, "advertise_addr")
2015-03-12 18:21:11 -04:00
}
// Pull out the cluster address since it's common to all backends
var clusterAddr string
if v, ok := m["cluster_addr"]; ok {
VAULT-33008: ipv6: always display RFC-5952 §4 conformant addresses (#29228) USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for IPv6-Only capabilities. This section requires that whenever Vault displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that _all_ IPv6 addresses must conform to RFC-5952 §4 text representation recommendations[2]. These recommendations do not prevent us from accepting RFC-4241[3] IPv6 addresses, however, whenever these same addresses are displayed they must conform to the strict RFC-5952 §4 guidelines. This PR implements handling of IPv6 address conformance in our `vault server` routine. We handle conformance normalization for all server, http_proxy, listener, seal, storage and telemetry configuration where an input could contain an IPv6 address, whether configured via an HCL file or via corresponding environment variables. The approach I've taken is to handle conformance normalization at parse time to ensure that all log output and subsequent usage inside of Vaults various subsystems always reference a conformant address, that way we don't need concern ourselves with conformance later. This approach ought to be backwards compatible to prior loose address configuration requirements, with the understanding that going forward all IPv6 representation will be strict regardless of what has been configured. In many cases I've updated our various parser functions to call the new `configutil.NormalizeAddr()` to apply conformance normalization. Others required no changes because they rely on standard library URL string output, which always displays IPv6 URLs in a conformant way. Not included in this changes is any other vault exec mode other than server. Client, operator commands, agent mode, proxy mode, etc. will be included in subsequent changes if necessary. [0]: https://www.nist.gov/publications/usgv6-profile [1]: https://www.nist.gov/publications/nist-ipv6-profile [2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4 [3]: https://www.rfc-editor.org/rfc/rfc4291 Signed-off-by: Ryan Cragun <me@ryan.ec>
2025-01-27 16:14:28 -05:00
clusterAddr = configutil.NormalizeAddr(v)
delete(m, "cluster_addr")
}
var disableClustering bool
var err error
if v, ok := m["disable_clustering"]; ok {
disableClustering, err = strconv.ParseBool(v)
if err != nil {
return multierror.Prefix(err, fmt.Sprintf("%s.%s:", name, key))
}
delete(m, "disable_clustering")
}
// Override with top-level values if they are set
if result.APIAddr != "" {
redirectAddr = configutil.NormalizeAddr(result.APIAddr)
}
if result.ClusterAddr != "" {
clusterAddr = configutil.NormalizeAddr(result.ClusterAddr)
}
if result.DisableClusteringRaw != nil {
disableClustering = result.DisableClustering
}
result.Storage = &Storage{
RedirectAddr: redirectAddr,
ClusterAddr: clusterAddr,
DisableClustering: disableClustering,
Type: strings.ToLower(key),
Config: m,
}
return nil
}
VAULT-33008: ipv6: always display RFC-5952 §4 conformant addresses (#29228) USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for IPv6-Only capabilities. This section requires that whenever Vault displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that _all_ IPv6 addresses must conform to RFC-5952 §4 text representation recommendations[2]. These recommendations do not prevent us from accepting RFC-4241[3] IPv6 addresses, however, whenever these same addresses are displayed they must conform to the strict RFC-5952 §4 guidelines. This PR implements handling of IPv6 address conformance in our `vault server` routine. We handle conformance normalization for all server, http_proxy, listener, seal, storage and telemetry configuration where an input could contain an IPv6 address, whether configured via an HCL file or via corresponding environment variables. The approach I've taken is to handle conformance normalization at parse time to ensure that all log output and subsequent usage inside of Vaults various subsystems always reference a conformant address, that way we don't need concern ourselves with conformance later. This approach ought to be backwards compatible to prior loose address configuration requirements, with the understanding that going forward all IPv6 representation will be strict regardless of what has been configured. In many cases I've updated our various parser functions to call the new `configutil.NormalizeAddr()` to apply conformance normalization. Others required no changes because they rely on standard library URL string output, which always displays IPv6 URLs in a conformant way. Not included in this changes is any other vault exec mode other than server. Client, operator commands, agent mode, proxy mode, etc. will be included in subsequent changes if necessary. [0]: https://www.nist.gov/publications/usgv6-profile [1]: https://www.nist.gov/publications/nist-ipv6-profile [2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4 [3]: https://www.rfc-editor.org/rfc/rfc4291 Signed-off-by: Ryan Cragun <me@ryan.ec>
2025-01-27 16:14:28 -05:00
// storageAddressKeys maps a storage backend type to its associated
// configuration whose values are URLs, IP addresses, or host:port style
// addresses. All physical storage types must have an entry in this map,
// otherwise our normalization check will fail when parsing the storage entry
// config. Physical storage types which don't contain such keys should include
// an empty array.
var storageAddressKeys = map[string][]string{
"aerospike": {"hostname"},
"alicloudoss": {"endpoint"},
"azure": {"arm_endpoint"},
"cassandra": {"hosts"},
"cockroachdb": {"connection_url"},
"consul": {"address", "service_address"},
"couchdb": {"endpoint"},
"dynamodb": {"endpoint"},
"etcd": {"address", "discovery_srv"},
"file": {},
"filesystem": {},
"foundationdb": {},
"gcs": {},
"inmem": {},
"inmem_ha": {},
"inmem_transactional": {},
"inmem_transactional_ha": {},
"manta": {"url"},
"mssql": {"server"},
"mysql": {"address"},
"oci": {},
"postgresql": {"connection_url"},
"raft": {}, // retry_join is handled separately in normalizeRaftRetryJoin()
"s3": {"endpoint"},
"spanner": {},
"swift": {"auth_url", "storage_url"},
"zookeeper": {"address"},
}
// normalizeStorageConfigAddresses takes a storage name, a configuration key
// and it's associated value and will normalize any URLs, IP addresses, or
// host:port style addresses.
func normalizeStorageConfigAddresses(storage string, key string, value string) (string, error) {
keys, ok := storageAddressKeys[storage]
if !ok {
return "", fmt.Errorf("unknown storage type %s", storage)
}
if slices.Contains(keys, key) {
return configutil.NormalizeAddr(value), nil
}
return value, nil
}
// normalizeRaftRetryJoin takes the hcl decoded value representation of a
// retry_join stanza and normalizes any URLs, IP addresses, or host:port style
// addresses, and returns the value encoded as JSON.
func normalizeRaftRetryJoin(val any) ([]byte, error) {
res := []map[string]any{}
// Depending on whether the retry_join stanzas were configured as an attribute,
// a block, or a mixture of both, we'll get different values from which we
// need to extract our individual retry joins stanzas.
stanzas := []map[string]any{}
if retryJoin, ok := val.([]map[string]any); ok {
// retry_join stanzas are defined as blocks
stanzas = retryJoin
} else {
// retry_join stanzas are defined as attributes or attributes and blocks
retryJoin, ok := val.([]any)
if !ok {
// retry_join stanzas have not been configured correctly
return nil, fmt.Errorf("malformed retry_join stanza: %v", val)
}
for _, stanza := range retryJoin {
stanzaVal, ok := stanza.(map[string]any)
if !ok {
return nil, fmt.Errorf("malformed retry_join stanza: %v", stanza)
}
stanzas = append(stanzas, stanzaVal)
}
}
for _, stanza := range stanzas {
normalizedStanza := map[string]any{}
for k, v := range stanza {
switch k {
case "auto_join":
cfg, err := discover.Parse(v.(string))
if err != nil {
return nil, err
}
for k, v := range cfg {
VAULT-33008: ipv6: always display RFC-5952 §4 conformant addresses (#29228) USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for IPv6-Only capabilities. This section requires that whenever Vault displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that _all_ IPv6 addresses must conform to RFC-5952 §4 text representation recommendations[2]. These recommendations do not prevent us from accepting RFC-4241[3] IPv6 addresses, however, whenever these same addresses are displayed they must conform to the strict RFC-5952 §4 guidelines. This PR implements handling of IPv6 address conformance in our `vault server` routine. We handle conformance normalization for all server, http_proxy, listener, seal, storage and telemetry configuration where an input could contain an IPv6 address, whether configured via an HCL file or via corresponding environment variables. The approach I've taken is to handle conformance normalization at parse time to ensure that all log output and subsequent usage inside of Vaults various subsystems always reference a conformant address, that way we don't need concern ourselves with conformance later. This approach ought to be backwards compatible to prior loose address configuration requirements, with the understanding that going forward all IPv6 representation will be strict regardless of what has been configured. In many cases I've updated our various parser functions to call the new `configutil.NormalizeAddr()` to apply conformance normalization. Others required no changes because they rely on standard library URL string output, which always displays IPv6 URLs in a conformant way. Not included in this changes is any other vault exec mode other than server. Client, operator commands, agent mode, proxy mode, etc. will be included in subsequent changes if necessary. [0]: https://www.nist.gov/publications/usgv6-profile [1]: https://www.nist.gov/publications/nist-ipv6-profile [2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4 [3]: https://www.rfc-editor.org/rfc/rfc4291 Signed-off-by: Ryan Cragun <me@ryan.ec>
2025-01-27 16:14:28 -05:00
// These are auto_join keys that are valid for the provider in go-discover
if slices.Contains([]string{"domain", "auth_url", "url", "host"}, k) {
cfg[k] = configutil.NormalizeAddr(v)
VAULT-33008: ipv6: always display RFC-5952 §4 conformant addresses (#29228) USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for IPv6-Only capabilities. This section requires that whenever Vault displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that _all_ IPv6 addresses must conform to RFC-5952 §4 text representation recommendations[2]. These recommendations do not prevent us from accepting RFC-4241[3] IPv6 addresses, however, whenever these same addresses are displayed they must conform to the strict RFC-5952 §4 guidelines. This PR implements handling of IPv6 address conformance in our `vault server` routine. We handle conformance normalization for all server, http_proxy, listener, seal, storage and telemetry configuration where an input could contain an IPv6 address, whether configured via an HCL file or via corresponding environment variables. The approach I've taken is to handle conformance normalization at parse time to ensure that all log output and subsequent usage inside of Vaults various subsystems always reference a conformant address, that way we don't need concern ourselves with conformance later. This approach ought to be backwards compatible to prior loose address configuration requirements, with the understanding that going forward all IPv6 representation will be strict regardless of what has been configured. In many cases I've updated our various parser functions to call the new `configutil.NormalizeAddr()` to apply conformance normalization. Others required no changes because they rely on standard library URL string output, which always displays IPv6 URLs in a conformant way. Not included in this changes is any other vault exec mode other than server. Client, operator commands, agent mode, proxy mode, etc. will be included in subsequent changes if necessary. [0]: https://www.nist.gov/publications/usgv6-profile [1]: https://www.nist.gov/publications/nist-ipv6-profile [2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4 [3]: https://www.rfc-editor.org/rfc/rfc4291 Signed-off-by: Ryan Cragun <me@ryan.ec>
2025-01-27 16:14:28 -05:00
}
}
normalizedStanza[k] = cfg.String()
VAULT-33008: ipv6: always display RFC-5952 §4 conformant addresses (#29228) USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for IPv6-Only capabilities. This section requires that whenever Vault displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that _all_ IPv6 addresses must conform to RFC-5952 §4 text representation recommendations[2]. These recommendations do not prevent us from accepting RFC-4241[3] IPv6 addresses, however, whenever these same addresses are displayed they must conform to the strict RFC-5952 §4 guidelines. This PR implements handling of IPv6 address conformance in our `vault server` routine. We handle conformance normalization for all server, http_proxy, listener, seal, storage and telemetry configuration where an input could contain an IPv6 address, whether configured via an HCL file or via corresponding environment variables. The approach I've taken is to handle conformance normalization at parse time to ensure that all log output and subsequent usage inside of Vaults various subsystems always reference a conformant address, that way we don't need concern ourselves with conformance later. This approach ought to be backwards compatible to prior loose address configuration requirements, with the understanding that going forward all IPv6 representation will be strict regardless of what has been configured. In many cases I've updated our various parser functions to call the new `configutil.NormalizeAddr()` to apply conformance normalization. Others required no changes because they rely on standard library URL string output, which always displays IPv6 URLs in a conformant way. Not included in this changes is any other vault exec mode other than server. Client, operator commands, agent mode, proxy mode, etc. will be included in subsequent changes if necessary. [0]: https://www.nist.gov/publications/usgv6-profile [1]: https://www.nist.gov/publications/nist-ipv6-profile [2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4 [3]: https://www.rfc-editor.org/rfc/rfc4291 Signed-off-by: Ryan Cragun <me@ryan.ec>
2025-01-27 16:14:28 -05:00
case "leader_api_addr":
normalizedStanza[k] = configutil.NormalizeAddr(v.(string))
default:
normalizedStanza[k] = v
}
}
res = append(res, normalizedStanza)
}
return json.Marshal(res)
}
func parseHAStorage(result *Config, list *ast.ObjectList, name string) error {
if len(list.Items) > 1 {
return fmt.Errorf("only one %q block is permitted", name)
2015-03-12 18:21:11 -04:00
}
// Get our item
item := list.Items[0]
key := name
if len(item.Keys) > 0 {
key = item.Keys[0].Token.Value().(string)
}
var config map[string]interface{}
if err := hcl.DecodeObject(&config, item.Val); err != nil {
return multierror.Prefix(err, fmt.Sprintf("%s.%s:", name, key))
}
m := make(map[string]string)
VAULT-33008: ipv6: always display RFC-5952 §4 conformant addresses (#29228) USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for IPv6-Only capabilities. This section requires that whenever Vault displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that _all_ IPv6 addresses must conform to RFC-5952 §4 text representation recommendations[2]. These recommendations do not prevent us from accepting RFC-4241[3] IPv6 addresses, however, whenever these same addresses are displayed they must conform to the strict RFC-5952 §4 guidelines. This PR implements handling of IPv6 address conformance in our `vault server` routine. We handle conformance normalization for all server, http_proxy, listener, seal, storage and telemetry configuration where an input could contain an IPv6 address, whether configured via an HCL file or via corresponding environment variables. The approach I've taken is to handle conformance normalization at parse time to ensure that all log output and subsequent usage inside of Vaults various subsystems always reference a conformant address, that way we don't need concern ourselves with conformance later. This approach ought to be backwards compatible to prior loose address configuration requirements, with the understanding that going forward all IPv6 representation will be strict regardless of what has been configured. In many cases I've updated our various parser functions to call the new `configutil.NormalizeAddr()` to apply conformance normalization. Others required no changes because they rely on standard library URL string output, which always displays IPv6 URLs in a conformant way. Not included in this changes is any other vault exec mode other than server. Client, operator commands, agent mode, proxy mode, etc. will be included in subsequent changes if necessary. [0]: https://www.nist.gov/publications/usgv6-profile [1]: https://www.nist.gov/publications/nist-ipv6-profile [2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4 [3]: https://www.rfc-editor.org/rfc/rfc4291 Signed-off-by: Ryan Cragun <me@ryan.ec>
2025-01-27 16:14:28 -05:00
for k, v := range config {
vStr, ok := v.(string)
if ok {
VAULT-33008: ipv6: always display RFC-5952 §4 conformant addresses (#29228) USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for IPv6-Only capabilities. This section requires that whenever Vault displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that _all_ IPv6 addresses must conform to RFC-5952 §4 text representation recommendations[2]. These recommendations do not prevent us from accepting RFC-4241[3] IPv6 addresses, however, whenever these same addresses are displayed they must conform to the strict RFC-5952 §4 guidelines. This PR implements handling of IPv6 address conformance in our `vault server` routine. We handle conformance normalization for all server, http_proxy, listener, seal, storage and telemetry configuration where an input could contain an IPv6 address, whether configured via an HCL file or via corresponding environment variables. The approach I've taken is to handle conformance normalization at parse time to ensure that all log output and subsequent usage inside of Vaults various subsystems always reference a conformant address, that way we don't need concern ourselves with conformance later. This approach ought to be backwards compatible to prior loose address configuration requirements, with the understanding that going forward all IPv6 representation will be strict regardless of what has been configured. In many cases I've updated our various parser functions to call the new `configutil.NormalizeAddr()` to apply conformance normalization. Others required no changes because they rely on standard library URL string output, which always displays IPv6 URLs in a conformant way. Not included in this changes is any other vault exec mode other than server. Client, operator commands, agent mode, proxy mode, etc. will be included in subsequent changes if necessary. [0]: https://www.nist.gov/publications/usgv6-profile [1]: https://www.nist.gov/publications/nist-ipv6-profile [2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4 [3]: https://www.rfc-editor.org/rfc/rfc4291 Signed-off-by: Ryan Cragun <me@ryan.ec>
2025-01-27 16:14:28 -05:00
var err error
m[k], err = normalizeStorageConfigAddresses(key, k, vStr)
if err != nil {
return err
}
continue
}
VAULT-33008: ipv6: always display RFC-5952 §4 conformant addresses (#29228) USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for IPv6-Only capabilities. This section requires that whenever Vault displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that _all_ IPv6 addresses must conform to RFC-5952 §4 text representation recommendations[2]. These recommendations do not prevent us from accepting RFC-4241[3] IPv6 addresses, however, whenever these same addresses are displayed they must conform to the strict RFC-5952 §4 guidelines. This PR implements handling of IPv6 address conformance in our `vault server` routine. We handle conformance normalization for all server, http_proxy, listener, seal, storage and telemetry configuration where an input could contain an IPv6 address, whether configured via an HCL file or via corresponding environment variables. The approach I've taken is to handle conformance normalization at parse time to ensure that all log output and subsequent usage inside of Vaults various subsystems always reference a conformant address, that way we don't need concern ourselves with conformance later. This approach ought to be backwards compatible to prior loose address configuration requirements, with the understanding that going forward all IPv6 representation will be strict regardless of what has been configured. In many cases I've updated our various parser functions to call the new `configutil.NormalizeAddr()` to apply conformance normalization. Others required no changes because they rely on standard library URL string output, which always displays IPv6 URLs in a conformant way. Not included in this changes is any other vault exec mode other than server. Client, operator commands, agent mode, proxy mode, etc. will be included in subsequent changes if necessary. [0]: https://www.nist.gov/publications/usgv6-profile [1]: https://www.nist.gov/publications/nist-ipv6-profile [2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4 [3]: https://www.rfc-editor.org/rfc/rfc4291 Signed-off-by: Ryan Cragun <me@ryan.ec>
2025-01-27 16:14:28 -05:00
var err error
var vBytes []byte
// Raft's retry_join requires special normalization due to its complexity
if key == "raft" && k == "retry_join" {
vBytes, err = normalizeRaftRetryJoin(v)
if err != nil {
return err
}
} else {
vBytes, err = json.Marshal(v)
if err != nil {
return err
}
}
VAULT-33008: ipv6: always display RFC-5952 §4 conformant addresses (#29228) USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for IPv6-Only capabilities. This section requires that whenever Vault displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that _all_ IPv6 addresses must conform to RFC-5952 §4 text representation recommendations[2]. These recommendations do not prevent us from accepting RFC-4241[3] IPv6 addresses, however, whenever these same addresses are displayed they must conform to the strict RFC-5952 §4 guidelines. This PR implements handling of IPv6 address conformance in our `vault server` routine. We handle conformance normalization for all server, http_proxy, listener, seal, storage and telemetry configuration where an input could contain an IPv6 address, whether configured via an HCL file or via corresponding environment variables. The approach I've taken is to handle conformance normalization at parse time to ensure that all log output and subsequent usage inside of Vaults various subsystems always reference a conformant address, that way we don't need concern ourselves with conformance later. This approach ought to be backwards compatible to prior loose address configuration requirements, with the understanding that going forward all IPv6 representation will be strict regardless of what has been configured. In many cases I've updated our various parser functions to call the new `configutil.NormalizeAddr()` to apply conformance normalization. Others required no changes because they rely on standard library URL string output, which always displays IPv6 URLs in a conformant way. Not included in this changes is any other vault exec mode other than server. Client, operator commands, agent mode, proxy mode, etc. will be included in subsequent changes if necessary. [0]: https://www.nist.gov/publications/usgv6-profile [1]: https://www.nist.gov/publications/nist-ipv6-profile [2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4 [3]: https://www.rfc-editor.org/rfc/rfc4291 Signed-off-by: Ryan Cragun <me@ryan.ec>
2025-01-27 16:14:28 -05:00
m[k] = string(vBytes)
}
// Pull out the redirect address since it's common to all backends
var redirectAddr string
if v, ok := m["redirect_addr"]; ok {
VAULT-33008: ipv6: always display RFC-5952 §4 conformant addresses (#29228) USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for IPv6-Only capabilities. This section requires that whenever Vault displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that _all_ IPv6 addresses must conform to RFC-5952 §4 text representation recommendations[2]. These recommendations do not prevent us from accepting RFC-4241[3] IPv6 addresses, however, whenever these same addresses are displayed they must conform to the strict RFC-5952 §4 guidelines. This PR implements handling of IPv6 address conformance in our `vault server` routine. We handle conformance normalization for all server, http_proxy, listener, seal, storage and telemetry configuration where an input could contain an IPv6 address, whether configured via an HCL file or via corresponding environment variables. The approach I've taken is to handle conformance normalization at parse time to ensure that all log output and subsequent usage inside of Vaults various subsystems always reference a conformant address, that way we don't need concern ourselves with conformance later. This approach ought to be backwards compatible to prior loose address configuration requirements, with the understanding that going forward all IPv6 representation will be strict regardless of what has been configured. In many cases I've updated our various parser functions to call the new `configutil.NormalizeAddr()` to apply conformance normalization. Others required no changes because they rely on standard library URL string output, which always displays IPv6 URLs in a conformant way. Not included in this changes is any other vault exec mode other than server. Client, operator commands, agent mode, proxy mode, etc. will be included in subsequent changes if necessary. [0]: https://www.nist.gov/publications/usgv6-profile [1]: https://www.nist.gov/publications/nist-ipv6-profile [2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4 [3]: https://www.rfc-editor.org/rfc/rfc4291 Signed-off-by: Ryan Cragun <me@ryan.ec>
2025-01-27 16:14:28 -05:00
redirectAddr = configutil.NormalizeAddr(v)
delete(m, "redirect_addr")
} else if v, ok := m["advertise_addr"]; ok {
VAULT-33008: ipv6: always display RFC-5952 §4 conformant addresses (#29228) USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for IPv6-Only capabilities. This section requires that whenever Vault displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that _all_ IPv6 addresses must conform to RFC-5952 §4 text representation recommendations[2]. These recommendations do not prevent us from accepting RFC-4241[3] IPv6 addresses, however, whenever these same addresses are displayed they must conform to the strict RFC-5952 §4 guidelines. This PR implements handling of IPv6 address conformance in our `vault server` routine. We handle conformance normalization for all server, http_proxy, listener, seal, storage and telemetry configuration where an input could contain an IPv6 address, whether configured via an HCL file or via corresponding environment variables. The approach I've taken is to handle conformance normalization at parse time to ensure that all log output and subsequent usage inside of Vaults various subsystems always reference a conformant address, that way we don't need concern ourselves with conformance later. This approach ought to be backwards compatible to prior loose address configuration requirements, with the understanding that going forward all IPv6 representation will be strict regardless of what has been configured. In many cases I've updated our various parser functions to call the new `configutil.NormalizeAddr()` to apply conformance normalization. Others required no changes because they rely on standard library URL string output, which always displays IPv6 URLs in a conformant way. Not included in this changes is any other vault exec mode other than server. Client, operator commands, agent mode, proxy mode, etc. will be included in subsequent changes if necessary. [0]: https://www.nist.gov/publications/usgv6-profile [1]: https://www.nist.gov/publications/nist-ipv6-profile [2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4 [3]: https://www.rfc-editor.org/rfc/rfc4291 Signed-off-by: Ryan Cragun <me@ryan.ec>
2025-01-27 16:14:28 -05:00
redirectAddr = configutil.NormalizeAddr(v)
delete(m, "advertise_addr")
}
// Pull out the cluster address since it's common to all backends
var clusterAddr string
if v, ok := m["cluster_addr"]; ok {
VAULT-33008: ipv6: always display RFC-5952 §4 conformant addresses (#29228) USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for IPv6-Only capabilities. This section requires that whenever Vault displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that _all_ IPv6 addresses must conform to RFC-5952 §4 text representation recommendations[2]. These recommendations do not prevent us from accepting RFC-4241[3] IPv6 addresses, however, whenever these same addresses are displayed they must conform to the strict RFC-5952 §4 guidelines. This PR implements handling of IPv6 address conformance in our `vault server` routine. We handle conformance normalization for all server, http_proxy, listener, seal, storage and telemetry configuration where an input could contain an IPv6 address, whether configured via an HCL file or via corresponding environment variables. The approach I've taken is to handle conformance normalization at parse time to ensure that all log output and subsequent usage inside of Vaults various subsystems always reference a conformant address, that way we don't need concern ourselves with conformance later. This approach ought to be backwards compatible to prior loose address configuration requirements, with the understanding that going forward all IPv6 representation will be strict regardless of what has been configured. In many cases I've updated our various parser functions to call the new `configutil.NormalizeAddr()` to apply conformance normalization. Others required no changes because they rely on standard library URL string output, which always displays IPv6 URLs in a conformant way. Not included in this changes is any other vault exec mode other than server. Client, operator commands, agent mode, proxy mode, etc. will be included in subsequent changes if necessary. [0]: https://www.nist.gov/publications/usgv6-profile [1]: https://www.nist.gov/publications/nist-ipv6-profile [2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4 [3]: https://www.rfc-editor.org/rfc/rfc4291 Signed-off-by: Ryan Cragun <me@ryan.ec>
2025-01-27 16:14:28 -05:00
clusterAddr = configutil.NormalizeAddr(v)
delete(m, "cluster_addr")
}
var disableClustering bool
var err error
if v, ok := m["disable_clustering"]; ok {
disableClustering, err = strconv.ParseBool(v)
if err != nil {
return multierror.Prefix(err, fmt.Sprintf("%s.%s:", name, key))
}
delete(m, "disable_clustering")
}
// Override with top-level values if they are set
if result.APIAddr != "" {
redirectAddr = configutil.NormalizeAddr(result.APIAddr)
}
if result.ClusterAddr != "" {
clusterAddr = configutil.NormalizeAddr(result.ClusterAddr)
}
if result.DisableClusteringRaw != nil {
disableClustering = result.DisableClustering
}
result.HAStorage = &Storage{
RedirectAddr: redirectAddr,
ClusterAddr: clusterAddr,
DisableClustering: disableClustering,
Type: strings.ToLower(key),
Config: m,
}
return nil
}
func parseServiceRegistration(result *Config, list *ast.ObjectList, name string) error {
if len(list.Items) > 1 {
return fmt.Errorf("only one %q block is permitted", name)
}
// Get our item
item := list.Items[0]
key := name
if len(item.Keys) > 0 {
key = item.Keys[0].Token.Value().(string)
}
var m map[string]string
if err := hcl.DecodeObject(&m, item.Val); err != nil {
return multierror.Prefix(err, fmt.Sprintf("%s.%s:", name, key))
}
VAULT-33008: ipv6: always display RFC-5952 §4 conformant addresses (#29228) USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for IPv6-Only capabilities. This section requires that whenever Vault displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that _all_ IPv6 addresses must conform to RFC-5952 §4 text representation recommendations[2]. These recommendations do not prevent us from accepting RFC-4241[3] IPv6 addresses, however, whenever these same addresses are displayed they must conform to the strict RFC-5952 §4 guidelines. This PR implements handling of IPv6 address conformance in our `vault server` routine. We handle conformance normalization for all server, http_proxy, listener, seal, storage and telemetry configuration where an input could contain an IPv6 address, whether configured via an HCL file or via corresponding environment variables. The approach I've taken is to handle conformance normalization at parse time to ensure that all log output and subsequent usage inside of Vaults various subsystems always reference a conformant address, that way we don't need concern ourselves with conformance later. This approach ought to be backwards compatible to prior loose address configuration requirements, with the understanding that going forward all IPv6 representation will be strict regardless of what has been configured. In many cases I've updated our various parser functions to call the new `configutil.NormalizeAddr()` to apply conformance normalization. Others required no changes because they rely on standard library URL string output, which always displays IPv6 URLs in a conformant way. Not included in this changes is any other vault exec mode other than server. Client, operator commands, agent mode, proxy mode, etc. will be included in subsequent changes if necessary. [0]: https://www.nist.gov/publications/usgv6-profile [1]: https://www.nist.gov/publications/nist-ipv6-profile [2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4 [3]: https://www.rfc-editor.org/rfc/rfc4291 Signed-off-by: Ryan Cragun <me@ryan.ec>
2025-01-27 16:14:28 -05:00
if key == "consul" {
if addr, ok := m["address"]; ok {
m["address"] = configutil.NormalizeAddr(addr)
}
}
result.ServiceRegistration = &ServiceRegistration{
Type: strings.ToLower(key),
Config: m,
}
return nil
}
// Sanitized returns a copy of the config with all values that are considered
// sensitive stripped. It also strips all `*Raw` values that are mainly
// used for parsing.
//
// Specifically, the fields that this method strips are:
// - Storage.Config
// - HAStorage.Config
// - Seals.Config
// - Telemetry.CirconusAPIToken
func (c *Config) Sanitized() map[string]interface{} {
// Create shared config if it doesn't exist (e.g. in tests) so that map
// keys are actually populated
if c.SharedConfig == nil {
c.SharedConfig = new(configutil.SharedConfig)
}
sharedResult := c.SharedConfig.Sanitized()
result := map[string]interface{}{
"cache_size": c.CacheSize,
"disable_sentinel_trace": c.DisableSentinelTrace,
"disable_cache": c.DisableCache,
"disable_printable_check": c.DisablePrintableCheck,
"enable_ui": c.EnableUI,
"max_lease_ttl": c.MaxLeaseTTL / time.Second,
"default_lease_ttl": c.DefaultLeaseTTL / time.Second,
"remove_irrevocable_lease_after": c.RemoveIrrevocableLeaseAfter / time.Second,
"cluster_cipher_suites": c.ClusterCipherSuites,
"plugin_directory": c.PluginDirectory,
"plugin_tmpdir": c.PluginTmpdir,
"plugin_file_uid": c.PluginFileUid,
"plugin_file_permissions": c.PluginFilePermissions,
"raw_storage_endpoint": c.EnableRawEndpoint,
"introspection_endpoint": c.EnableIntrospectionEndpoint,
"api_addr": c.APIAddr,
"cluster_addr": c.ClusterAddr,
"disable_clustering": c.DisableClustering,
"disable_performance_standby": c.DisablePerformanceStandby,
"disable_sealwrap": c.DisableSealWrap,
"disable_indexing": c.DisableIndexing,
"allow_audit_log_prefixing": c.AllowAuditLogPrefixing,
"enable_response_header_hostname": c.EnableResponseHeaderHostname,
"enable_response_header_raft_node_id": c.EnableResponseHeaderRaftNodeID,
"log_requests_level": c.LogRequestsLevel,
"experiments": c.Experiments,
"enable_unauthenticated_access": c.EnableUnauthenticatedAccess,
"detect_deadlocks": c.DetectDeadlocks,
"imprecise_lease_role_tracking": c.ImpreciseLeaseRoleTracking,
"enable_post_unseal_trace": c.EnablePostUnsealTrace,
"post_unseal_trace_directory": c.PostUnsealTraceDir,
}
for k, v := range sharedResult {
result[k] = v
}
// Sanitize storage stanza
if c.Storage != nil {
storageType := c.Storage.Type
sanitizedStorage := map[string]interface{}{
"type": storageType,
"redirect_addr": c.Storage.RedirectAddr,
"cluster_addr": c.Storage.ClusterAddr,
"disable_clustering": c.Storage.DisableClustering,
}
if storageType == "raft" {
sanitizedStorage["raft"] = map[string]interface{}{
"max_entry_size": c.Storage.Config["max_entry_size"],
}
for k, v := range c.Storage.Config {
sanitizedStorage["raft"].(map[string]interface{})[k] = v
}
}
result["storage"] = sanitizedStorage
}
// Sanitize observations stanza
if c.Observations != nil {
sanitizedObservations := map[string]interface{}{
"ledger_path": c.Observations.LedgerPath,
}
result["observations"] = sanitizedObservations
}
// Sanitize HA storage stanza
if c.HAStorage != nil {
haStorageType := c.HAStorage.Type
sanitizedHAStorage := map[string]interface{}{
"type": haStorageType,
"redirect_addr": c.HAStorage.RedirectAddr,
"cluster_addr": c.HAStorage.ClusterAddr,
"disable_clustering": c.HAStorage.DisableClustering,
}
if haStorageType == "raft" {
sanitizedHAStorage["raft"] = map[string]interface{}{
"max_entry_size": c.HAStorage.Config["max_entry_size"],
}
}
result["ha_storage"] = sanitizedHAStorage
}
// Sanitize service_registration stanza
if c.ServiceRegistration != nil {
sanitizedServiceRegistration := map[string]interface{}{
"type": c.ServiceRegistration.Type,
}
result["service_registration"] = sanitizedServiceRegistration
}
entConfigResult := c.entConfig.Sanitized()
for k, v := range entConfigResult {
result[k] = v
}
return result
}
func (c *Config) Prune() {
for _, l := range c.Listeners {
l.RawConfig = nil
l.UnusedKeys = nil
}
c.FoundKeys = nil
c.UnusedKeys = nil
c.SharedConfig.FoundKeys = nil
c.SharedConfig.UnusedKeys = nil
if c.Telemetry != nil {
c.Telemetry.FoundKeys = nil
c.Telemetry.UnusedKeys = nil
}
}
func (c *Config) found(s, k string) {
delete(c.UnusedKeys, s)
c.FoundKeys = append(c.FoundKeys, k)
}
func (c *Config) ToVaultNodeConfig() (*testcluster.VaultNodeConfig, error) {
var vnc testcluster.VaultNodeConfig
err := mapstructure.Decode(c, &vnc)
if err != nil {
return nil, err
}
return &vnc, nil
}