FETCH_ALL.md
# Secrets Provider - Fetch All
## Existing Functionality
In the regular configuration of Secrets Provider, the application developer must
specify each secret that needs to be retrieved from Conjur. This is done in one
of two ways, depending on how the secrets are being provided:
### Kubernetes Secrets
```yaml
apiVersion: v1
kind: Secret
metadata:
name: test-app-secrets-provider-k8s-secret
type: Opaque
stringData:
conjur-map: |-
DB_URL: test-secrets-provider-k8s-app-db/url
DB_USERNAME: test-secrets-provider-k8s-app-db/username
DB_PASSWORD: test-secrets-provider-k8s-app-db/password
```
The values in the K8s secret can now be used in the application pod like so:
```yaml
annotations:
conjur.org/secrets-destination: k8s_secrets
conjur.org/k8s-secrets: |
- test-app-secrets-provider-k8s-secret
...
env:
- name: DB_USERNAME
valueFrom:
secretKeyRef:
name: test-app-secrets-provider-k8s-secret
key: DB_USERNAME
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: test-app-secrets-provider-k8s-secret
key: DB_PASSWORD
```
### Push to File
```yaml
annotations:
conjur.org/secrets-destination: file
conjur.org/conjur-secrets.test-app: |
- db-url: test-secrets-provider-p2f-app-db/url
- admin-username: test-secrets-provider-p2f-app-db/username
- admin-password: test-secrets-provider-p2f-app-db/password
conjur.org/secret-file-path.test-app: "./application.yaml"
conjur.org/secret-file-format.test-app: "yaml"
```
## New Functionality
With the introduction of the Fetch All feature, the application developer can
now retrieve all secrets that the host has access to using the following new
syntax:
### Kubernetes Secrets
```yaml
apiVersion: v1
kind: Secret
metadata:
name: test-k8s-secret-fetch-all
type: Opaque
stringData:
# Now choose ONE of the following two options:
# For plaintext secrets
conjur-map: |-
"*": "*"
# For secrets that should be decoded from Base64
conjur-map: |-
"*":
id: "*"
content-type: base64
```
The values in the K8s secret can now be used in the application pod in the usual
way, referencing the secret path as the key in the secret map, while replacing
any `/` with `.` (see [Limitations](#aliases-and-key-names) for more details).
```yaml
annotations:
conjur.org/secrets-destination: k8s_secrets
conjur.org/k8s-secrets: |
- test-k8s-secret-fetch-all
...
env:
- name: FETCH_ALL_TEST_SECRETS
valueFrom:
secretKeyRef:
name: test-k8s-secret-fetch-all
key: test-secrets-provider-k8s-app-db.username
```
Alternatively, you can use the following syntax to push all the retrieved secrets into
environment variables:
```yaml
envFrom:
- secretRef:
name: test-k8s-secret-fetch-all
```
This will create an environment variable for each secret, for example:
```sh
test-secrets-provider-k8s-app-db.url="..."
test-secrets-provider-k8s-app-db.username="..."
test-secrets-provider-k8s-app-db.password="..."
```
### Push to File
```yaml
conjur.org/secrets-destination: file
conjur.org/secret-file-format.test-app: json # or "yaml", or "template"
# Add any other relevant annotations, such as "conjur.org/secret-file-path.test-app", here
# Now choose ONE of the following two options:
# For plaintext secrets
conjur.org/conjur-secrets.test-app: "*"
# For secrets that should be decoded from Base64
conjur.org/conjur-secrets.group8: |
- "*": "*"
content-type: base64
```
With this configuration, the Secrets Provider will retrieve all secrets that the
host has access to and provide them to the application pod in the usual way.
## Limitations
There are several important things to note about this feature:
### Security Implications
- The Fetch All feature should be used with caution, as it can expose more
secrets than intended. Ideally, Conjur should be configured to only allow the
host to access the secrets that it needs. Additionally, the host should only
be used for a small unit, such as a single application, and therefore only
need access to a small number of secrets. If the host has access to a large
number of secrets, it may be a sign that the host is too permissive and should
be restricted. In the case of a compromised host, the attacker would have
access to all secrets that the host has access to, which could be a
significant security risk. It is important to follow the principle of least
privilege in general, and even more so when using the Fetch All feature.
### Performance and Reliability
- Using Fetch All will be slightly slower than specifying each secret
individually, as it requires multiple requests to Conjur - first to list all
the available secrets, and then to fetch them. In cases where performance is
critical, it may be better to specify the secrets individually.
- Using Fetch All introduces a certain amount of unpredictability into the
application, as the set of secrets that the host has access to may change,
which could cause the application to behave unexpectedly. When specifying the
secrets individually, Secrets Provider will fail if any of the specified
secrets are not available, which can be a useful signal that something is
wrong. With Fetch All, the application will continue to run even if some
secrets are missing, and the application may behave in unexpected ways as a
result. If using Fetch All, ensure that the application will handle missing
secrets gracefully.
- To prevent denial of service due to very large numbers of secrets, the maximum
number of secrets supported is 500. Secrets Provider will cease fetching secrets
once it reaches this limit and log an error with code `CSPFK010D`.
### Aliases and Key Names
- There is no way to use aliases for secrets when using the Fetch All feature.
This means that the keys used for the secrets (both in K8s Secrets and P2F)
will be the *full path* of the secret in Conjur. At the same time, Kubernetes
secrets do not allow keys to contain slashes (`/`) or most other special
characters. Due to these limitations:
- *In K8s secrets mode:* Any slashes, spaces or other special characters
(besides `_`, `-`, and `.`) in the Conjur secret path will be replaced with
dots (`.`) in the key names when using K8s Secrets. For example, if the
secret is stored at `host/my-app/secrets/db-password`, the key in the K8s
Secret will be `host.my-app.secrets.db-password`.
- *Duplicate keys:* If there are two or more secrets that, after this
character replacement, have the same key, a warning will be logged with
error code `CSPFK067E`. The first secret will be used, and the others will
be ignored. For example, if there are two secrets at
`host/my-app/secrets/db.password` and `host/my-app/secrets/db password`
(with a space in place of a `.`), the key in the K8s Secret for each of them
will be `host.my-app.secrets.db.password`. The order in which the secrets
are processed is not guaranteed, so one of them will be used and the other
will be ignored. This will cause non-deterministic behavior in the
application and must be avoided.
- *In P2F mode:* The key names will be the full path of the secret in Conjur.
For example, if the secret is stored at `host/my-app/secrets/db-password`,
the key in the P2F file will be `host/my-app/secrets/db-password`.
This poses no issues for YAML and JSON files, since those formats
support special characters in key names. However, it is impossible to
use the `bash`, `dotenv`, and `properties` formats with secrets that have
`/` in their path, and they are therefore not supported for use with the
Fetch All feature. Additionally, when using custom templates, it must be
ensured that the key names are valid for the chosen format.
- *Custom Templates:* When using custom templates, aside from the above
restriction due to keys containing `/`, it is also not possible to directly
reference the secret path in the template. This is because the secret path
is not known at the time the template is validated, before the secrets are
actually retrieved. Additionally, there may be different secrets returned
for subsequent fetches of the same template (when using rotation).
Therefore, custom templates cannot rely on specific secret paths, and must
instead use the `SecretsArray` variable to iterate over all secrets. Here is
an example template that prints all fetched secrets in Base64 encoding:
```yaml
conjur.org/secret-file-template.test-app: |
{{range .SecretsArray}}{{ .Alias }}: {{ .Value | b64enc }}{{ "\n" }}{{end}}
```
The resulting file will look like this:
```txt
host/my-app/secrets/db-username: YWRtaW4=
host/my-app/secrets/db-password: cGFzc3dvcmQ=
```
### Summary
The Fetch All feature is a powerful tool that can simplify the configuration of
Secrets Provider by allowing the application developer to retrieve all secrets
that the host has access to with a single configuration. However, it should be
used with caution, as it can expose more secrets than intended, and has some
limitations that must be taken into account when using it.
**For these reasons, we recommend using the Fetch All feature only in cases where
it is impractical to specify each secret individually, and only after carefully
considering the security implications and limitations of the feature.**