Vault
Custom Secrets Engine

Define a Configuration for the Secrets Engine

  • 15min

  • Vault
  • Video

In the Define a backend for the secrets engine tutorial, you configured the secrets engine to retrieve attributes and create a new client to access your target API.

In this tutorial, you will create a configuration schema for your secrets engine. The configuration will include attributes passed through the config path of the secrets engine.

Step 2 defines a secrets engine configuration at /secrets-engine/config

To do this, you will:

  1. Set up your development environment.
    You will clone the HashiCups secrets engine repository. This contains many of the interfaces and objects you need to create a secrets engine.
  2. Define the fields for the secrets engine's configuration.
    You will define a set of fields that a Vault operator passes to configure the secrets engine.
  3. Implement read for the secrets engine's configuration.
    You will implement a method to handle reading the configuration from the secrets engine backend.
  4. Implement create and update for the secrets engine's configuration.
    You will implement a method to handle writing the configuration to the secrets engine backend.
  5. Implement delete for the secrets engine's configuration.
    You will implement a method to handle deletion of the configuration from the secrets engine backend.
  6. Add the configuration path to the backend.
    You will update the backend to add a new API path for the configuration.
  7. Explore unit tests that verify the configuration path.
    You will examine unit tests that check that Vault can create, read, update, and delete the secrets engine configuration.

Prerequisites

Note: Complete the tutorial to define the backend for the secrets engine.

Set up your development environment

Clone the vault-guides repository.

$ git clone https://github.com/hashicorp/vault-guides

Change into the plugin development directory for the HashiCups secrets engine.

$ cd plugins/vault-plugin-secrets-hashicups

Note: If you are stuck in this tutorial, refer to the plugins/vault-plugin-secrets-hashicups/solution directory.

Define the fields for the secrets engine's configuration

Open path_config.go. The file contains all of the objects and methods related to setting up the config path for the secrets engine.

Note: Replace the methods and structs in the scaffold with the embedded code examples.

The file should already include implementation for the following:

  • configStoragePath: defines the config path for the secrets engine's configuration.

  • hashiCupsConfig: defines a configuration object with username, password, and URL to the target API (HashiCups).

  • getConfig: passes the context and storage path for Vault to store the hashiCupsConfig into the config path for the secrets engine.

However, the pathConfig method returns an object with empty fields. These attributes defined framework.Path extend the Vault API for the secrets engine's config path.

Replace the pathConfig method in the scaffold.

path_config.go
func pathConfig(b *hashiCupsBackend) *framework.Path {
    return &framework.Path{
        Pattern: "config",
        Fields: map[string]*framework.FieldSchema{
            "username": {
                Type:        framework.TypeString,
                Description: "The username to access HashiCups Product API",
                Required:    true,
                DisplayAttrs: &framework.DisplayAttributes{
                    Name:      "Username",
                    Sensitive: false,
                },
            },
            "password": {
                Type:        framework.TypeString,
                Description: "The user's password to access HashiCups Product API",
                Required:    true,
                DisplayAttrs: &framework.DisplayAttributes{
                    Name:      "Password",
                    Sensitive: true,
                },
            },
            "url": {
                Type:        framework.TypeString,
                Description: "The URL for the HashiCups Product API",
                Required:    true,
                DisplayAttrs: &framework.DisplayAttributes{
                    Name:      "URL",
                    Sensitive: false,
                },
            },
        },
        Operations: map[logical.Operation]framework.OperationHandler{},
        ExistenceCheck:  b.pathConfigExistenceCheck,
        HelpSynopsis:    pathConfigHelpSynopsis,
        HelpDescription: pathConfigHelpDescription,
    }
}

You must define the Fields that extend the Vault config endpoint. They must match the attributes in hashiCupsConfig. Each field attribute must include the following:

  1. Type: All of the attributes for hashiCupsConfig use Type.String. You can find a list of types in the Vault SDK documentation.
  2. Description: Purpose of the attribute
  3. Required: Used for OpenAPI output. Whether or not a Vault operator must define the attribute when configuring the secrets engine. You must enforce it in the handler of your secrets engine, the schema will not enforce it for you!
  4. DisplayAttr: Used for OpenAPI output. Includes name and whether or not it should be output based on the value in Sensitive.
path_config.go
1 2 3 4 5 6 7 8 9 1011121314151617181920212223242526272829303132333435363738func pathConfig(b *hashiCupsBackend) *framework.Path {
    return &framework.Path{
        Pattern: "config",
        Fields: map[string]*framework.FieldSchema{
            "username": {
                Type:        framework.TypeString,
                Description: "The username to access HashiCups Product API",
                Required:    true,
                DisplayAttrs: &framework.DisplayAttributes{
                    Name:      "Username",
                    Sensitive: false,
                },
            },
            "password": {
                Type:        framework.TypeString,
                Description: "The user's password to access HashiCups Product API",
                Required:    true,
                DisplayAttrs: &framework.DisplayAttributes{
                    Name:      "Password",
                    Sensitive: true,
                },
            },
            "url": {
                Type:        framework.TypeString,
                Description: "The URL for the HashiCups Product API",
                Required:    true,
                DisplayAttrs: &framework.DisplayAttributes{
                    Name:      "URL",
                    Sensitive: false,
                },
            },
        },
        Operations: map[logical.Operation]framework.OperationHandler{},
        ExistenceCheck:  b.pathConfigExistenceCheck,
        HelpSynopsis:    pathConfigHelpSynopsis,
        HelpDescription: pathConfigHelpDescription,
    }
}

The pathConfig includes additional attributes for the configuration, such as ExistenceCheck. ExistenceCheck calls the pathConfigExistenceCheck function, which verifies whether or not the configuration already exists in Vault.

Note: Adding an ExistenceCheck will affect your Vault access control list (ACL) policy for the secrets engine. When you define the ExistenceCheck for the configuration, a Vault operator setting up the secrets engine must have the create capability to add the configuration.

path_config.go
1 2 3 4 5 6 7 8 9 1011121314151617181920212223242526272829303132333435363738394041424344454647func pathConfig(b *hashiCupsBackend) *framework.Path {
    return &framework.Path{
        Pattern: "config",
        Fields: map[string]*framework.FieldSchema{
            "username": {
                Type:        framework.TypeString,
                Description: "The username to access HashiCups Product API",
                Required:    true,
                DisplayAttrs: &framework.DisplayAttributes{
                    Name:      "Username",
                    Sensitive: false,
                },
            },
            "password": {
                Type:        framework.TypeString,
                Description: "The user's password to access HashiCups Product API",
                Required:    true,
                DisplayAttrs: &framework.DisplayAttributes{
                    Name:      "Password",
                    Sensitive: true,
                },
            },
            "url": {
                Type:        framework.TypeString,
                Description: "The URL for the HashiCups Product API",
                Required:    true,
                DisplayAttrs: &framework.DisplayAttributes{
                    Name:      "URL",
                    Sensitive: false,
                },
            },
        },
        Operations: map[logical.Operation]framework.OperationHandler{},
        ExistenceCheck:  b.pathConfigExistenceCheck,
        HelpSynopsis:    pathConfigHelpSynopsis,
        HelpDescription: pathConfigHelpDescription,
    }
}

func (b *hashiCupsBackend) pathConfigExistenceCheck(ctx context.Context, req *logical.Request, data *framework.FieldData) (bool, error) {
    out, err := req.Storage.Get(ctx, req.Path)
    if err != nil {
        return false, fmt.Errorf("existence check failed: %w", err)
    }

    return out != nil, nil
}

The path includes attributes for help text, such as HelpSynopsis and HelpDescription. The help text describes the attributes someone needs to configure the secrets engine.

path_config.go
1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031323334353637383940414243444546474849func pathConfig(b *hashiCupsBackend) *framework.Path {
    return &framework.Path{
        Pattern: "config",
        Fields: map[string]*framework.FieldSchema{
            "username": {
                Type:        framework.TypeString,
                Description: "The username to access HashiCups Product API",
                Required:    true,
                DisplayAttrs: &framework.DisplayAttributes{
                    Name:      "Username",
                    Sensitive: false,
                },
            },
            "password": {
                Type:        framework.TypeString,
                Description: "The user's password to access HashiCups Product API",
                Required:    true,
                DisplayAttrs: &framework.DisplayAttributes{
                    Name:      "Password",
                    Sensitive: true,
                },
            },
            "url": {
                Type:        framework.TypeString,
                Description: "The URL for the HashiCups Product API",
                Required:    true,
                DisplayAttrs: &framework.DisplayAttributes{
                    Name:      "URL",
                    Sensitive: false,
                },
            },
        },
        Operations: map[logical.Operation]framework.OperationHandler{},
        ExistenceCheck:  b.pathConfigExistenceCheck,
        HelpSynopsis:    pathConfigHelpSynopsis,
        HelpDescription: pathConfigHelpDescription,
    }
}

const pathConfigHelpSynopsis = `Configure the HashiCups backend.`

const pathConfigHelpDescription = `
The HashiCups secret backend requires credentials for managing
JWTs issued to users working with the products API.

You must sign up with a username and password and
specify the HashiCups address for the products API
before using this secrets engine backend.
`

Implement read for the secrets engine's configuration

Open path_config.go. The file contains all of the objects and methods related to setting up the config path for the secrets engine.

Note: Replace the methods and structs in the scaffold with the embedded code examples.

The Operations field in the pathConfig method starts empty. You need to add methods to the Operations field to tell Vault how to handle creating, reading, updating, and deleting information at the config path.

Note: When you build a secrets engine and define its configuration, you need to implement operations to read, create, update and delete information at each API path you define for the secrets engine.

Create a new method named pathConfigRead in path_config.go. The method reads the configuration and outputs non-sensitive fields, specifically the HashiCups username and URL.

path_config.go
func (b *hashiCupsBackend) pathConfigRead(ctx context.Context, req *logical.Request, data *framework.FieldData) (*logical.Response, error) {
    config, err := getConfig(ctx, req.Storage)
    if err != nil {
        return nil, err
    }

    return &logical.Response{
        Data: map[string]interface{}{
            "username": config.Username,
            "url":      config.URL,
        },
    }, nil
}

Under the Operations field, add logical.ReadOperation to the list of OperationHandler and callback to pathConfigRead. The secrets engine responds to a read operation from Vault with this method.

path_config.go
1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930313233343536373839404142func pathConfig(b *hashiCupsBackend) *framework.Path {
    return &framework.Path{
        Pattern: "config",
        Fields: map[string]*framework.FieldSchema{
            "username": {
                Type:        framework.TypeString,
                Description: "The username to access HashiCups Product API",
                Required:    true,
                DisplayAttrs: &framework.DisplayAttributes{
                    Name:      "Username",
                    Sensitive: false,
                },
            },
            "password": {
                Type:        framework.TypeString,
                Description: "The user's password to access HashiCups Product API",
                Required:    true,
                DisplayAttrs: &framework.DisplayAttributes{
                    Name:      "Password",
                    Sensitive: true,
                },
            },
            "url": {
                Type:        framework.TypeString,
                Description: "The URL for the HashiCups Product API",
                Required:    true,
                DisplayAttrs: &framework.DisplayAttributes{
                    Name:      "URL",
                    Sensitive: false,
                },
            },
        },
        Operations: map[logical.Operation]framework.OperationHandler{
            logical.ReadOperation: &framework.PathOperation{
                Callback: b.pathConfigRead,
            },
        },
        ExistenceCheck:  b.pathConfigExistenceCheck,
        HelpSynopsis:    pathConfigHelpSynopsis,
        HelpDescription: pathConfigHelpDescription,
    }
}

Implement create and update for the secrets engine's configuration

Open path_config.go. The file contains all of the objects and methods related to setting up the config path for the secrets engine.

Note: Replace the methods and structs in the scaffold with the embedded code examples.

Create a new method named pathConfigWrite in path_config.go. The method accounts for specific logic, including:

  • If the configuration does not exist and you need to update it, the handler throws an error.
  • Verify you passed a username, URL, and password for the target API to the configuration using the GetOk method. You use GetOk to enforce required attributes during a CreateOperation.
  • Write the new or updated configuration using Storage.Put.
  • Reset the configuration so Vault picks up the new configuration.
path_config.go
func (b *hashiCupsBackend) pathConfigWrite(ctx context.Context, req *logical.Request, data *framework.FieldData) (*logical.Response, error) {
    config, err := getConfig(ctx, req.Storage)
    if err != nil {
        return nil, err
    }

    createOperation := (req.Operation == logical.CreateOperation)

    if config == nil {
        if !createOperation {
            return nil, errors.New("config not found during update operation")
        }
        config = new(hashiCupsConfig)
    }

    if username, ok := data.GetOk("username"); ok {
        config.Username = username.(string)
    } else if !ok && createOperation {
        return nil, fmt.Errorf("missing username in configuration")
    }

    if url, ok := data.GetOk("url"); ok {
        config.URL = url.(string)
    } else if !ok && createOperation {
        return nil, fmt.Errorf("missing url in configuration")
    }

    if password, ok := data.GetOk("password"); ok {
        config.Password = password.(string)
    } else if !ok && createOperation {
        return nil, fmt.Errorf("missing password in configuration")
    }

    entry, err := logical.StorageEntryJSON(configStoragePath, config)
    if err != nil {
        return nil, err
    }

    if err := req.Storage.Put(ctx, entry); err != nil {
        return nil, err
    }

    b.reset()

    return nil, nil
}

Under the Operations field, add logical.CreateOperation and logical.UpdateOperation to the list of OperationHandler. Both operations should callback to pathConfigWrite.

path_config.go
1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930313233343536373839404142434445464748func pathConfig(b *hashiCupsBackend) *framework.Path {
    return &framework.Path{
        Pattern: "config",
        Fields: map[string]*framework.FieldSchema{
            "username": {
                Type:        framework.TypeString,
                Description: "The username to access HashiCups Product API",
                Required:    true,
                DisplayAttrs: &framework.DisplayAttributes{
                    Name:      "Username",
                    Sensitive: false,
                },
            },
            "password": {
                Type:        framework.TypeString,
                Description: "The user's password to access HashiCups Product API",
                Required:    true,
                DisplayAttrs: &framework.DisplayAttributes{
                    Name:      "Password",
                    Sensitive: true,
                },
            },
            "url": {
                Type:        framework.TypeString,
                Description: "The URL for the HashiCups Product API",
                Required:    true,
                DisplayAttrs: &framework.DisplayAttributes{
                    Name:      "URL",
                    Sensitive: false,
                },
            },
        },
        Operations: map[logical.Operation]framework.OperationHandler{
            logical.ReadOperation: &framework.PathOperation{
                Callback: b.pathConfigRead,
            },
            logical.CreateOperation: &framework.PathOperation{
                Callback: b.pathConfigWrite,
            },
            logical.UpdateOperation: &framework.PathOperation{
                Callback: b.pathConfigWrite,
            },
        },
        ExistenceCheck:  b.pathConfigExistenceCheck,
        HelpSynopsis:    pathConfigHelpSynopsis,
        HelpDescription: pathConfigHelpDescription,
    }
}

Note: For most secrets engines, you can consolidate handling for create and update of fields. Ensure that the method runs idempotently (repeatedly running the function does not change the fields, unless you make changes).

Implement delete for the secrets engine's configuration

Open path_config.go. The file contains all of the objects and methods related to setting up the config path for the secrets engine.

Note: Replace the methods and structs in the scaffold with the embedded code examples.

Create a new method named pathConfigDelete in path_config.go. The method deletes the configuration from the secrets engine backend and resets the secrets engine.

path_config.go
123456789func (b *hashiCupsBackend) pathConfigDelete(ctx context.Context, req *logical.Request, data *framework.FieldData) (*logical.Response, error) {
    err := req.Storage.Delete(ctx, configStoragePath)

    if err == nil {
        b.reset()
    }

    return nil, err
}

Under the Operations field, add logical.DeleteOperation to the list of OperationHandler. It should callback to pathConfigDelete.

path_config.go
1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930313233343536373839404142434445464748495051func pathConfig(b *hashiCupsBackend) *framework.Path {
    return &framework.Path{
        Pattern: "config",
        Fields: map[string]*framework.FieldSchema{
            "username": {
                Type:        framework.TypeString,
                Description: "The username to access HashiCups Product API",
                Required:    true,
                DisplayAttrs: &framework.DisplayAttributes{
                    Name:      "Username",
                    Sensitive: false,
                },
            },
            "password": {
                Type:        framework.TypeString,
                Description: "The user's password to access HashiCups Product API",
                Required:    true,
                DisplayAttrs: &framework.DisplayAttributes{
                    Name:      "Password",
                    Sensitive: true,
                },
            },
            "url": {
                Type:        framework.TypeString,
                Description: "The URL for the HashiCups Product API",
                Required:    true,
                DisplayAttrs: &framework.DisplayAttributes{
                    Name:      "URL",
                    Sensitive: false,
                },
            },
        },
        Operations: map[logical.Operation]framework.OperationHandler{
            logical.ReadOperation: &framework.PathOperation{
                Callback: b.pathConfigRead,
            },
            logical.CreateOperation: &framework.PathOperation{
                Callback: b.pathConfigWrite,
            },
            logical.UpdateOperation: &framework.PathOperation{
                Callback: b.pathConfigWrite,
            },
            logical.DeleteOperation: &framework.PathOperation{
                Callback: b.pathConfigDelete,
            },
        },
        ExistenceCheck:  b.pathConfigExistenceCheck,
        HelpSynopsis:    pathConfigHelpSynopsis,
        HelpDescription: pathConfigHelpDescription,
    }
}

Add the configuration path to the backend

For each API path you extend on the secrets engine, you must add it to the secrets engine backend.

Note: Replace the methods and structs in the scaffold with the embedded code examples.

Open backend.go and replace backend to add pathConfig to the list of valid paths for the backend.

backend.go
1 2 3 4 5 6 7 8 9 1011121314151617181920212223func backend() *hashiCupsBackend {
    var b = hashiCupsBackend{}

    b.Backend = &framework.Backend{
        Help: strings.TrimSpace(backendHelp),
        PathsSpecial: &logical.Paths{
            LocalStorage: []string{},
            SealWrapStorage: []string{
                "config",
                "role/*",
            },
        },
        Paths: framework.PathAppend(
            []*framework.Path{
                pathConfig(&b),
            },
        ),
        Secrets: []*framework.Secret{},
        BackendType: logical.TypeLogical,
        Invalidate:  b.invalidate,
    }
    return &b
}

Note: If you do not add your path to the backend object, you will get an error of unsupported path in your tests and compiled plugin.

Explore unit tests that verify the configuration path

The Vault Plugin SDK includes a testing framework for unit and acceptance tests.

  • Unit tests: Use mocks to verify the functionality of the secrets engine
  • Acceptance tests: Require a Vault instance, an active target API endpoint, and binary for the secrets engine.

You can write a set of unit tests to pass in fields and mock the Vault backend. The tests verify secrets engine creates, reads, updates, and deletes the configuration.

Open backend_test.go and examine getTestBackend. The method mocks a backend object using the Vault Plugin SDK. It references the interfaces in the HashiCups backend Factory.

backend_test.go
1 2 3 4 5 6 7 8 9 101112131415func getTestBackend(tb testing.TB) (*hashiCupsBackend, logical.Storage) {
    tb.Helper()

    config := logical.TestBackendConfig()
    config.StorageView = new(logical.InmemStorage)
    config.Logger = hclog.NewNullLogger()
    config.System = logical.TestSystemView()

    b, err := Factory(context.Background(), config)
    if err != nil {
        tb.Fatal(err)
    }

    return b.(*hashiCupsBackend), config.StorageView
}

Note: You can reuse getTestBackend for your own secrets engine. Return your secrets engine's backend object instead of hashiCupsBackend.

Open path_config_test.go. The file includes a set of constants that you will pass as configuration fields for the config path. The HashiCups configuration requires username, password, and URL. The unit tests will not issue requests to the API endpoint.

path_config_test.go
const (
    username = "vault-plugin-testing"
    password = "Testing!123"
    url      = "http://localhost:19090"
)

Examine the method TestConfig. It creates the mock backend with getTestBackend and runs a series of tests creating, reading, updating, and deleting the configuration using the constants.

path_config_test.go
func TestConfig(t *testing.T) {
    b, reqStorage := getTestBackend(t)

    t.Run("Test Configuration", func(t *testing.T) {
        err := testConfigCreate(t, b, reqStorage, map[string]interface{}{
            "username": username,
            "password": password,
            "url":      url,
        })

        assert.NoError(t, err)

        err = testConfigRead(t, b, reqStorage, map[string]interface{}{
            "username": username,
            "url":      url,
        })

        assert.NoError(t, err)

        err = testConfigUpdate(t, b, reqStorage, map[string]interface{}{
            "username": username,
            "url":      "http://hashicups:19090",
        })

        assert.NoError(t, err)

        err = testConfigRead(t, b, reqStorage, map[string]interface{}{
            "username": username,
            "url":      "http://hashicups:19090",
        })

        assert.NoError(t, err)

        err = testConfigDelete(t, b, reqStorage)

        assert.NoError(t, err)
    })
}

Examine testConfigCreate as an example. It calls the mock backend with a logical.CreateOperation at the path, config. The data includes the configuration fields defined as constants, such as username, password, and URL.

path_config_test.go
1 2 3 4 5 6 7 8 9 1011121314151617func testConfigCreate(t *testing.T, b logical.Backend, s logical.Storage, d map[string]interface{}) error {
    resp, err := b.HandleRequest(context.Background(), &logical.Request{
        Operation: logical.CreateOperation,
        Path:      configStoragePath,
        Data:      d,
        Storage:   s,
    })

    if err != nil {
        return err
    }

    if resp != nil && resp.IsError() {
        return resp.Error()
    }
    return nil
}

TestConfig runs the tests sequentially and passes the same storage object between tests. You should write your test sequence as follows:

  1. Create the configuration.
  2. Read the configuration to test if the create succeeded.
  3. Update the configuration.
  4. Read the configuration to test if the update succeeded.
  5. Delete the configuration.
  6. Check for errors.

Open a terminal and make sure your working directory uses the plugins/vault-plugin-secrets-hashicups.

$ pwd

${HOME}/hashicorp/vault-guides/plugins/vault-plugin-secrets-hashicups

Run the configuration path tests in your terminal. The tests should pass.

$ go test -v -run TestConfig

=== RUN   TestConfig
=== RUN   TestConfig/Test_Configuration
--- PASS: TestConfig (0.00s)
    --- PASS: TestConfig/Test_Configuration (0.00s)
PASS
ok      github.com/hashicorp/vault-guides/plugins/vault-plugin-secrets-hashicups        0.186s

Next steps

Congratulations! You added the config path to your secrets engine.

If you are stuck in this tutorial, refer to the plugins/vault-plugin-secrets-hashicups/solution directory.

Previous
Next