Vault
Troubleshoot KMIP secrets engine with Oracle TDE
Introduction
When you integrate Vault's KMIP secrets engine with Oracle Database Transparent Data Encryption (TDE), you can meet with issues which prevent the solution from functioning.
Most of the issues you can experience have root causes which fall into 4 main categories:
File access: Vault integration with Oracle Database TDE involves installing a PKCS#11 library file. Issues can arise around the path, permissions, or with Linux security systems such as SELinux or AppArmor.
Network communication: Typical network issues including connectivity between Vault and Oracle Database, network port availability, and firewall configuration can all contribute to problems with the integration.
KMIP certificates: Common issues include problems with the certificate format, certificate chain of trust, or registration.
Misconfiguration: Problems with Vault configuration, Oracle Database PKCS#11 HCL configuration file, or the
sqlnet.orafile.
This guidance can help you to identify the root cause of a problem with your KMIP secrets engine integration, and resolve the problem.
Essential resources
If you are not familiar with troubleshooting the Vault server or secrets engines, review Troubleshoot Vault for general Vault troubleshooting guidance before exploring the issues and resolutions.
About the command examples
The command examples in this resource are illustrative starting points, and might not function for you as-is. You need to update the commands, and replace critical example values like file names, paths, addresses, host names, and other values specific to your deployment with your actual values.
Troubleshoot KMIP secrets engine
If you run into trouble integrating the Vault KMIP secrets engine with Oracle TDE, use these Vault observability resources to help you troubleshoot the secrets engine.
Vault operational log output is commonly collected by the systemd journal on Linux servers, and you can typically find these logs in your organization's log aggregation solution. You can retrieve errors directly from the journal with a command like this example.
$ journalctl -xe -u vaultBe aware that as of version 1.13.0, Vault also supports logging to a static file, so you might need to check the Vault server configuration to determine if your deployment uses this file instead of the systemd journal.
Note
Vault log output defaults to the "info" level, which might not capture sufficient detail to help find an issue root cause. You can increase the log verbosity levels as described in the server documentation.
Vault API or CLI error messages: when you enable and configure the KMIP secrets engine with the Vault API or CLI, Vault emits helpful warning and error messages when problems occur. Use these messages in combination with Vault server operational log messages to isolate and resolve the issue root cause.
You can further enhance your troubleshooting efforts by enabling some helpful debug options.
If you operate Vault on Linux with systemd, you can enable any or all debug options by adding these environment variables to /etc/vault.d/vault.env followed by a restart of the Vault service:
- KMIP_CONNECTION_DEBUGGING=1 enables debug output for connectivity related operations.
- KMIP_REQUEST_DEBUGGING=1 enables debug output for requests.
- KMIP_RESPONSE_DEBUGGING=1 enables debug output for responses.
- KMIP_DECODING_DEBUGGING=1 enables debug output for operations related to decoding messages.
- KMIP_INDEX_DEBUGGING=1 enables debug output for message processing and indexing operations.
- KMIP_DUMP_BYTE_VALUES=1 enables debug output containing raw byte values.
If your Vault deployment is not on a systemd based Linux, you can export the necessary environment variables in the process manager or shell that Vault starts from. Here are examples for every debug variable:
KMIP_CONNECTION_DEBUGGING=1
KMIP_REQUEST_DEBUGGING=1
KMIP_RESPONSE_DEBUGGING=1
KMIP_DECODING_DEBUGGING=1
KMIP_INDEX_DEBUGGING=1
KMIP_DUMP_BYTE_VALUES=1
Troubleshoot Oracle Database
You can find indicators of issues as warnings or errors emitted from the following resources for Oracle Database.
- The Oracle Database client emits messages in with a related code in the format:
ORA-NNNNN Message text. - The Oracle Database alert log file in
$ORACLE_BASE/diag/rdbms/<DB_NAME>/<INSTANCE>/trace/alert_<INSTANCE>.log - The Oracle Database trace files in
$ORACLE_BASE/diag/rdbms/<DB_NAME>/<INSTANCE>/trace/
Check these log file sources for PKCS or wallet errors.
Use this reference of Oracle Database error codes related to issues which can arise when integrating the Wallet and Vault KMIP secrets engine.
| Error Code | Description | Common Causes |
|---|---|---|
| ORA-28365 | Wallet is not open | Wallet not opened, auto-login failed, PKCS#11 library not accessible |
| ORA-46633 | Cannot create Master Key | Vault KMIP secrets engine server unreachable, authentication failed, insufficient permissions |
| ORA-46658 | Could not activate Master Key | Key not found in Vault, key ID mismatch |
| ORA-46693 | PKCS#11 library not found | Incorrect path in sqlnet.ora, library file missing |
| ORA-46694 | Cannot load PKCS#11 library | Permission issues, wrong architecture, missing dependencies |
| ORA-46695 | PKCS#11 module returned error | Generic PKCS#11 error, check library logs and Vault audit logs |
| ORA-46696 | PKCS#11 configuration file invalid | vault-pkcs11.hcl syntax error, certificate format issues |
| ORA-28374 | Typed master key not found | Key mismatch between Oracle and Vault, key rotation issue |
Explore common issues and resolutions
Use the general troubleshooting workflows and resources to narrow the issue scope to a given category of issue. After narrowing the issue scope, you can use the details in each corresponding tab to learn more about specific issue scenarios, along with their troubleshooting and resolution steps.
These are common issues with Oracle Database access the Vault PKCS#11 provider file, and their resolution steps.
Note
Ensure that you update the path shown in the example commands to match the path to the library file in your Oracle Database deployment.
Library file not found
If Oracle Database cannot find the library file, the result is errors from Oracle Database or the OS.
Oracle Database error messages
ORA-28365: wallet is not open
ORA-46693: The PKCS#11 library specified in the external security module configuration is not found
Root cause
Oracle cannot locate the Vault PKCS#11 library file specified in sqlnet.ora, or the library path is not correct.
Resolution steps
Verify that the library exists.
$ ls -la /opt/oracle/extapi/64/hsm/vault/0.2.1/libvault-pkcs11.soCheck if the file is a symbolic link pointing to the correct file location.
$ file /opt/oracle/extapi/64/hsm/vault/0.2.1/libvault-pkcs11.soUpdate sqlnet.ora with the correct path.
ENCRYPTION_WALLET_LOCATION= (SOURCE=(METHOD=PKCS11) (METHOD_DATA= (DIRECTORY=/opt/oracle/wallet) (PKCS11_LIBRARY=/opt/oracle/extapi/64/hsm/vault/0.2.1/libvault-pkcs11.so)))Verify Oracle user has read access to the directory path.
Library file ownership or permissions
Ownership or permission issues on the PKCS#11 library file result in errors from Oracle Database or the OS.
Oracle Database error message
ORA-46694: The PKCS#11 library cannot be loaded
OS error message
Permission denied
Root cause
Insufficient file ownership, permissions, or security restrictions preventing Oracle Database from loading the library.
Troubleshooting steps
Confirm that the Oracle Database user (default: 'oracle') is the owner, and has read and execute permissions (-rwxr-xr-x or 0755) on the library file.
$ stat /opt/oracle/extapi/64/hsm/vault/0.2.1/libvault-pkcs11.so
File: /opt/oracle/extapi/64/hsm/vault/0.2.1/libvault-pkcs11.so
Size: 26917768 Blocks: 52576 IO Block: 4096 regular file
Device: 29h/41d Inode: 1067402 Links: 1
Access: (0755/-rwxr-xr-x) Uid: (54321/ oracle) Gid: (54321/oinstall)
Access: 2024-10-03 19:12:45.000000000 +0000
Modify: 2024-10-03 19:12:45.000000000 +0000
Change: 2025-10-08 15:01:06.170653846 +0000
Birth: 2025-10-08 14:52:35.921493613 +0000
If you use SELinux, check for appropriate Oracle context on the library file.
$ ls -Z /opt/oracle/extapi/64/hsm/vault/0.2.1/libvault-pkcs11.so
Resolution steps
Set correct ownership.
$ chown oracle:oinstall /opt/oracle/extapi/64/hsm/vault/0.2.1/libvault-pkcs11.soSet correct permissions.
$ chmod 755 /opt/oracle/extapi/64/hsm/vault/0.2.1/libvault-pkcs11.soIf you use SELinux, check if it is enforcing. If SELinux is enforcing, check that the appropriate Oracle context is present for the library file or consider adding a permanent policy for the file. Instructions for this step are dependent on your installation. You must review the examples shown here before you apply changes based on your deployment.
Check if SELinux is enforcing.
$ getenforceSet appropriate context on the file.
$ chcon -t textrel_shlib_t /opt/oracle/extapi/64/hsm/vault/0.2.1/libvault-pkcs11.soAdd a permanent policy for the file.
$ semanage fcontext -a -t textrel_shlib_t \ "/opt/oracle/extapi/64/hsm/vault/0.2.1/libvault-pkcs11.so"Restore default contexts on file.
$ restorecon -v /opt/vault/lib/libvault-kmip.soIf you use AppArmor, you may need to add an exception to Oracle's AppArmor profile.
Check AppArmor status:
$ aa-statusEdit
/etc/apparmor.d/local/oracleand add:/opt/oracle/extapi/64/hsm/vault/0.2.1/libvault-pkcs11.so r, /opt/oracle/extapi/64/hsm/vault/0.2.1/libvault-pkcs11.so m,Reload AppArmor:
$ systemctl reload apparmor
Library missing dependencies
Oracle Database error message
ORA-46694: The PKCS#11 library cannot be loaded
OS error message
error while loading shared libraries: libcrypto.so.1.1: cannot open shared object file
Root cause
The PKCS#11 library has dependencies not found in the system's library path.
Resolution steps
Check library dependencies, and look for any "not found" entries.
$ ldd /opt/oracle/extapi/64/hsm/vault/0.2.1/libvault-pkcs11.so linux-vdso.so.1 (0x000073d73cbd5000) libdl.so.2 => /lib64/libdl.so.2 (0x000073d73b074000) libpthread.so.0 => /lib64/libpthread.so.0 (0x000073d73ae54000) libresolv.so.2 => /lib64/libresolv.so.2 (0x000073d73ac3c000) libc.so.6 => /lib64/libc.so.6 (0x000073d73a865000) /lib64/ld-linux-x86-64.so.2 (0x000073d73c800000)Install any missing dependencies using the OS package manager.
Update the
LD_LIBRARY_PATHenvironment variable if necessary to add the path to the library file or create a library configuration file.
Wrong library file architecture or OS
If the PKCS#11 library file does not match the system architecture or OS, the result is errors from Oracle Database or the OS.
Oracle Database error message
ORA-46694: The PKCS#11 library cannot be loaded
OS error message
wrong ELF class: ELFCLASS32
Root cause
These issues result from architecture or OS mismatch between the library file and the host system.
Resolution steps
Verify Oracle Database host system architecture.
$ uname -a Linux f8f6a2d69670 6.14.0-1014-gcp #15~24.04.1-Ubuntu SMP Fri Jul 25 23:26:08 UTC 2025 x86_64 x86_64 x86_64 GNU/LinuxVerify that the library architecture matches the host architecture.
$ file /opt/oracle/extapi/64/hsm/vault/0.2.1/libvault-pkcs11.so libvault-pkcs11.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=4240b9491a1fb466f31b91bda11ea9740cfe1abc, with debug_info, not strippedDownload and use the correct library file version.
Functional end-to-end test workflow
This is a basic functional test workflow to check for a problem free integration of the KMIP secrets engine with Oracle Database.
Test network connectivity from the Oracle Database host to the Vault KMIP server listener host with a tool like netcat (
nc).$ nc -zv vault.example.com 5696Test TLS handshake from Oracle Database server to Vault KMIP server listener host with
openssl.$ openssl s_client -connect vault.example.com:5696 \ -cert /opt/oracle/wallet/client-cert.pem \ -key /opt/oracle/wallet/client-key.pem \ -CAfile /opt/oracle/wallet/ca.pem \ -showcertsExpected output snippet:
...snip... SSL handshake has read ... bytes SSL handshake has sent ... bytes Verify return code: 0 (ok) ...snip...Verify Oracle Database can load the PKCS#11 client library with a database query like this example.
SELECT * FROM V$ENCRYPTION_WALLET;Test key creation with this database query.
ADMINISTER KEY MANAGEMENT SET ENCRYPTION KEY IDENTIFIED BY "C0ff33_1s@great!";Encrypt a test tablespace.
CREATE TABLESPACE encrypted_ts DATAFILE '/u01/app/oracle/oradata/orcl/encrypted01.dbf' SIZE 100M ENCRYPTION USING 'AES256' DEFAULT STORAGE(ENCRYPT);
Troubleshooting checklist
When you define a checklist or runbook to handle issues, you benefit from having a focused set of steps with outcomes you can use to help resolve issues.
You can use this checklist as a starting point for defining your own integration troubleshooting steps.
| Checked | Component | Condition |
|---|---|---|
| Vault server | KMIP secrets engine running and listening on port TCP/5696 | |
| Vault server | KMIP scope and role properly configured | |
| PKCS#11 | Library file exists at the specified path | |
| PKCS#11 | Library file has correct ownership and permissions | |
| PKCS#11 | Library file is for the correct architecture and OS | |
| PKCS#11 | AppArmor or SELinux permits access to library file | |
| Oracle Database | Connectivity to Vault server on port TCP/5696 | |
| Oracle Database | Wallet directory exists with correct ownership and permissions | |
| Oracle Database | Oracle process user can read certificate files specified in PKCS#11 HCL configuration | |
| Oracle Database | PKCS#11 HCL configuration file is valid | |
| Oracle Database | PKCS#11 HCL configuration file specifies correct certificate paths | |
| Certificates | Client certificate is valid and not expired | |
| Certificates | Client certificate registered in correct Vault KMIP role scope | |
| Certificates | Client certificate files in correct (PEM) format | |
| Certificates | Certificate Authority certificate chain is valid and complete | |
| Certificates | If using wallet auto-login feature, ensure client certificate's private key has no password set. |
Summary
You learned about troubleshooting the Vault KMIP secrets engine integration with Oracle Database TDE through a wide range of examples. The examples represent most common types of issues along with their troubleshooting and resolution steps.
You now have resources, and tools like common error codes and a troubleshooting checklist starter to enhance your troubleshooting workflows.