Chapter 1: Cryptography, PKI Structure & Fundamentals

Before You Begin: This chapter builds the conceptual foundation you need before touching a single certificate. By the end, you will understand why cryptography exists, how it works at a practical level, and what happens behind the scenes when two machines establish a secure connection.

1.1 Why Use Cryptography?

Imagine sending a postcard. Anyone who handles it—postal workers, neighbors, strangers—can read it. Now imagine the postcard contains your bank password. That is what unencrypted network traffic looks like.

Every packet traveling across a network can be intercepted, read, modified, or forged. Without cryptography:

Cryptography solves all four of these problems. It is not optional in modern systems—it is the foundation of every secure communication.

1.2 The Four Pillars of Information Security

Cryptography provides four fundamental guarantees. Every secure system relies on a combination of these:

Confidentiality — “Only you can read this”

Confidentiality ensures that data is readable only by the intended recipient. Even if an attacker intercepts the data, they see only meaningless noise.

How it is implemented:

• Symmetric encryption (AES-256): Same key encrypts and decrypts. Fast, used for bulk data.
• Asymmetric encryption (RSA, ECC): Public key encrypts, private key decrypts. Used for key exchange.

What it protects against: Eavesdropping.

Integrity — “This has not been changed”

Integrity guarantees that data has not been altered between sender and receiver. If even a single bit changes, the modification is detected.

How it is implemented:

• Hash functions (SHA-256): Produce a fixed-size fingerprint of data.
• HMAC: Hash combined with a secret key for authenticated integrity.
• Digital signatures: Hash signed with a private key.

 Original:  "Transfer $100 to Bob"  → SHA-256 → a1b2c3d4...
 Tampered:  "Transfer $900 to Bob"  → SHA-256 → f7e8d9c0...  ← DIFFERENT!

What it protects against: Tampering.

Authenticity — “You are who you say you are”

Authenticity proves the identity of the communicating party. When you connect to your bank’s website, you need assurance that it is actually your bank, not an impersonator.

How it is implemented:

• Digital certificates (X.509): Bind a public key to an identity.
• Certificate Authorities (CAs): Trusted third parties that verify identities.
• Digital signatures: Prove that a message was created by the claimed sender.

What it protects against: Impersonation.

Non-Repudiation — “You cannot deny this”

Non-repudiation ensures that the sender cannot deny having sent a message or performed an action. This is the digital equivalent of a handwritten signature on a contract.

How it is implemented:

• Digital signatures with private keys: Only the key holder can produce the signature.
• Timestamping: Proves when an action occurred.
• Audit logs with cryptographic integrity: Tamper-evident records.

What it protects against: Repudiation (denying responsibility).

Summary: The Four Pillars

1.3 Hash Functions: Digital Fingerprints

A hash function takes an input of any size and produces an output of fixed size. Think of it as a fingerprint for data.

Properties and Examples

1. Deterministic — Same input always produces the same output.

SHA-256("Hello") → 185f8db32271...  (always)
SHA-256("Hello") → 185f8db32271...  (always)

2. One-Way (Pre-image Resistance) — You cannot reverse-engineer the input from the output.

185f8db32271... → ???  (computationally infeasible to find input)

3. Collision Resistant — It is practically impossible to find two different inputs that produce the same output.

SHA-256("input A") → hash1
SHA-256("input B") → hash2
hash1 ≠ hash2  (with overwhelming probability)

4. Avalanche Effect — A tiny change in input produces a completely different output.

SHA-256("Hello World")  → a591a6d40bf420404a011733cfb7b190...
SHA-256("Hello World!") → 7f83b1657ff1fc53b92dc18148a1d65d...
                                     ↑ completely different!

Are Hashes Safe?

Not all hash algorithms are equal. Some have been broken:

“Broken” means: An attacker can find two different inputs that produce the same hash (collision). This allows forging documents, certificates, or signatures.

# Verify for yourself — compute hashes on any RHEL system:
echo -n "Hello World" | sha256sum
# a591a6d40bf420404a011733cfb7b190d62c65bf0bcda32b57b277d9ad9f146e

echo -n "Hello World" | md5sum
# b10a8db164e0754105b7a99be72e3fe5  ← DO NOT trust this for security!

Real-World Uses of Hashes

1.4 Symmetric vs Asymmetric Cryptography

Symmetric: One Key for Everything

Both sender and receiver share the same secret key. Like a padlock where both parties have a copy of the same key.

Asymmetric: Two Keys, Two Roles

Each party has a key pair: a public key (share freely) and a private key (never share).

Why We Need Both: Hybrid Encryption

Asymmetric cryptography solves the key distribution problem, but it is too slow for bulk data. The solution: use asymmetric to exchange a symmetric key, then use symmetric for the actual data.

This is exactly what happens in every HTTPS connection.

1.5 Understanding Key Exchange: The Color Mixing Analogy

Before diving into the real TLS/RSA handshake, let’s build intuition with a visual analogy. This explains Diffie-Hellman key exchange, the mechanism used in modern TLS to establish a shared secret.

The Problem

Alice and Bob want to agree on a shared secret color that Eve (the eavesdropper) cannot figure out, even though Eve can see everything they send each other.

Why Eve Cannot Cheat

Mixing paint is easy to do but impossible to reverse. You cannot separate mixed paint back into its components. In mathematics, this is analogous to:

• Easy: Multiply two large primes → get a product (mixing)
• Hard: Factor a large product → find the primes (unmixing)

This is the one-way function that makes cryptography work.

From Colors to Numbers

1.6 The TLS Handshake: How a Secure Connection Really Works

Now let’s see what actually happens when your browser connects to https://bank.com. This combines everything we’ve learned: hashes, asymmetric crypto, symmetric crypto, certificates, and key exchange.

Step-by-Step Walkthrough

Step 1-2 (Hello): Client and server exchange capabilities and random numbers. These randoms add freshness—they ensure every session is unique even between the same parties.

Step 3 (Certificate): The server proves its identity by sending its X.509 certificate containing its public key.

Step 4 (Verification): This is the critical trust step. The client walks the chain of trust:

Each signature is verified using the issuer’s public key. If any link breaks, the handshake fails.

Step 5 (Key Exchange): The client generates 48 bytes of random data (PreMasterSecret), encrypts it with the server’s public RSA key, and sends it. Only the server’s private key can decrypt this—this is the asymmetric magic.

Step 6 (Key Derivation): Both sides independently compute the same session keys using a Pseudo-Random Function (PRF). This is where we transition from slow asymmetric to fast symmetric.

Step 7-9 (Encrypted Communication): From here on, everything is encrypted with AES-256—thousands of times faster than RSA.

Modern TLS 1.3: Simpler and Faster

TLS 1.3 simplified the handshake by removing RSA key exchange (forward secrecy is now mandatory) and reducing round trips:

1.7 PKI Structure: The Trust Architecture

Public Key Infrastructure (PKI) is the system that manages digital certificates and public keys. It answers the question: “How do I know this public key really belongs to bank.com?”

Why a Chain?

Root CAs are extremely valuable—if compromised, every certificate they ever signed becomes untrusted. So Root CAs are:

• Stored in offline, air-gapped hardware security modules (HSMs)
• Used only to sign Intermediate CA certificates
• Valid for 20-30 years

Intermediate CAs do the daily work of issuing certificates. If compromised:

• Only that Intermediate’s certificates are affected
• The Root can revoke the Intermediate and create a new one
• Damage is contained

Revocation: What Happens When Trust Breaks

When a private key is compromised or a certificate should no longer be trusted:

1.8 Putting It All Together: A Complete Example

Let’s trace a complete HTTPS connection from start to finish, seeing every concept in action:

1.9 Key Takeaways

Before moving on to RHEL-specific certificate management, make sure you understand:

Chapter Navigation

Chapter 2: Introduction to Certificates on RHEL

Welcome! This tutorial will take you from knowing nothing about digital certificates to confidently troubleshooting certificate issues on Red Hat Enterprise Linux systems.

2.1 Why This Tutorial?

You’re a RHEL administrator. One day, something breaks:

• Apache refuses to start: SSL_CTX_use_certificate:ca md too weak
• LDAP connections fail: TLS: hostname does not match CN
• certmonger shows: CA_UNREACHABLE
• curl returns: SSL certificate problem: unable to get local issuer certificate

Sound familiar? These are certificate issues, and they’re everywhere in modern Linux systems.

This tutorial teaches you to:

• ✅ Understand what certificates are (RHEL perspective)
• ✅ Configure certificates for common RHEL services
• ✅ Troubleshoot certificate problems (primary goal!)
• ✅ Automate certificate lifecycle with RHEL tools
• ✅ Handle RHEL version differences (7, 8, 9, 10)
• ✅ Pass audits (FIPS, STIG, compliance)

2.2 Who Is This For?

Primary Audience:

• RHEL administrators and engineers
• Support engineers troubleshooting certificate issues
• Anyone managing RHEL systems with HTTPS, LDAPS, or TLS

Prerequisites:

• Basic Linux command line knowledge
• Access to RHEL systems (7, 8, 9, or 10)
• No prior certificate knowledge needed!

2.3 What Are Certificates? (In 60 Seconds)

Imagine you visit https://example.com. How does your browser know it’s really talking to example.com and not an imposter?

Answer: Digital certificates.

A certificate is like a digital ID card that:

1. Proves identity (“I am example.com”)
2. Enables encryption (secure communication)
3. Is signed by a trusted authority (like a CA)

On RHEL Systems

Certificates are used everywhere:

• Web servers (Apache, NGINX) → HTTPS
• Directory services (OpenLDAP, FreeIPA) → LDAPS
• Mail servers (Postfix, Dovecot) → SMTPS/IMAPS
• Databases (PostgreSQL, MySQL) → TLS connections
• APIs and services (REST, microservices) → mTLS
• VPN tunnels → Secure connections
• Container registries → Secure images

Bottom line: If it’s networked and secure on RHEL, it probably uses certificates.

2.4 Your First Certificate Inspection

Let’s get hands-on immediately. On any RHEL system, run:

# Check a web server's TLS certificate (HTTPS, port 443)
echo | openssl s_client -connect access.redhat.com:443 2>/dev/null | openssl x509 -noout -text | head -20

# Alternative: inspect mail STARTTLS on port 25 (-starttls smtp matches SMTP, not SSH)
echo | openssl s_client -connect localhost:25 -starttls smtp 2>/dev/null | openssl x509 -noout -text | head -20

You’ll see output like:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0a:5d:d2:48:fc:4e:2f:e2:99:81:09:74:2d:4c:d5:69
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: C=US, O=DigiCert Inc, CN=DigiCert Global G3 TLS ECC SHA384 2020 CA1
        Validity
            Not Before: Oct 30 00:00:00 2025 GMT
            Not After : Oct 27 23:59:59 2026 GMT
        Subject: C=US, ST=North Carolina, L=Raleigh, O=Red Hat, Inc., CN=access.redhat.com
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:fc:08:bf:d2:d8:63:0c:84:a4:c8:dd:04:9c:8c:
                    99:4f:cb:93:31:7f:9e:64:27:ea:3d:a7:18:fd:3e:
                    4c:c2:58:8b:cb:f2:5c:6e:95:bf:f3:97:ba:b8:2b:
                    49:c6:51:30:f4:71:88:e3:fa:d4:f1:73:74:1d:e3:
                    2b:49:bc:9e:6e

What you’re seeing:

• Issuer: Who signed this certificate
• Subject: Who this certificate belongs to
• Validity: When it’s valid (Not Before / Not After dates)
• Signature Algorithm: How it’s secured (for example, ECDSA with SHA-384)

🎉 Congratulations! You just inspected your first certificate.

2.5 How Certificates Work (RHEL Context)

The Three Key Components

1. Certificate (.crt, .pem)

   • Public information: “I am server.example.com”
   • Contains the public key
   • Stored in /etc/pki/tls/certs/ on RHEL

2. Private Key (.key, .pem)

   • Secret! Never share this
   • Used to prove you own the certificate
   • Stored in /etc/pki/tls/private/ on RHEL (mode 600!)

3. Certificate Authority (CA)

   • Issues and signs certificates
   • Could be public (Let’s Encrypt, DigiCert)
   • Or internal (FreeIPA, corporate CA)
   • Trusted CAs stored in /etc/pki/ca-trust/ on RHEL

The Trust Chain

Root CA (trusted by RHEL system)
  └─ Intermediate CA
      └─ Server Certificate (your web server)

When someone connects to your RHEL server:

1. Server sends its certificate
2. Client verifies signature chain back to a trusted root CA
3. If chain is valid → connection proceeds
4. If chain breaks → error (and you get the support call!)

2.6 RHEL’s Certificate Architecture

Key Directories

/etc/pki/
├── ca-trust/
│   ├── source/anchors/      ← Put custom CA certs here
│   └── extracted/           ← System-wide trust store
│       ├── pem/             ← PEM format CAs
│       ├── openssl/         ← OpenSSL trust
│       └── java/            ← Java trust (cacerts)
├── tls/
│   ├── certs/               ← Server certificates
│   ├── private/             ← Private keys (mode 700!)
│   └── cert.pem             ← Default certificate symlink
└── nssdb/                   ← NSS database (Firefox, etc.)

Key Tools

# OpenSSL - Swiss army knife of certificates
openssl version  # Check your version

# NSS Tools - For NSS databases
certutil -L -d /etc/pki/nssdb

# Trust Management - Add/remove CAs
update-ca-trust  # RHEL's trust store updater

# Certificate Manager - Automatic renewal (RHEL 7+)
getcert list  # Show tracked certificates

# Crypto-Policies - System-wide security (RHEL 8+)
update-crypto-policies --show  # Check current policy

2.7 A Day in the Life: Certificate Scenarios

Scenario 1: Adding a Custom CA

# You have a corporate CA that signed your internal servers
sudo cp corporate-ca.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust extract

# Now RHEL trusts certificates signed by your corporate CA!

Scenario 2: Setting Up Apache HTTPS

# Install Apache with SSL/TLS
sudo dnf install httpd mod_ssl

# Generate a private key
sudo openssl genpkey -algorithm RSA -out /etc/pki/tls/private/server.key \
  -pkeyopt rsa_keygen_bits:2048

# Generate a certificate signing request (CSR)
sudo openssl req -new -key /etc/pki/tls/private/server.key \
  -out /tmp/server.csr \
  -subj "/CN=web.example.com"

# Send CSR to CA, get certificate back, install it
sudo cp server.crt /etc/pki/tls/certs/

# Configure Apache, restart
sudo systemctl restart httpd

Scenario 3: Troubleshooting an Expired Certificate

# Service fails with: "certificate has expired"
# Check certificate expiration
sudo openssl x509 -in /etc/pki/tls/certs/server.crt -noout -dates

# Output shows:
# notAfter=Jan 15 23:59:59 2024 GMT  ← Oops, expired!

# Renew certificate, replace file, restart service

2.8 RHEL Version Differences (Preview)

Certificate management has evolved significantly across RHEL versions:

Don’t worry! Chapter 8 covers these version differences in detail.

2.9 Common Certificate Problems (Preview)

You’ll learn to troubleshoot:

Configuration Issues:

• Certificate/key mismatch
• Wrong file permissions
• Incorrect paths in config files

Trust Issues:

• Self-signed certificates rejected
• Unknown CA errors
• Chain validation failures

Expiration Issues:

• Expired certificates
• Clock skew problems
• Renewal failures

Version Issues:

• TLS version mismatches
• Cipher suite problems
• Crypto-policy conflicts (RHEL 8+)
• OpenSSL 3.x compatibility (RHEL 9+)

Service-Specific:

• Apache: SSLCertificateFile errors
• NGINX: ssl_certificate problems
• Postfix: TLS handshake failures
• LDAP: TLS: hostname does not match

2.10 Learning Path Overview

This tutorial is organized for RHEL administrators:

Part 1: Foundations (Chapters 1-7)

Start here. Learn certificate basics in RHEL context.

Part 2: Version-Specific (Chapters 8-13)

Deep dive into RHEL 7, 8, 9, 10 differences.

Part 3: Services (Chapters 14-21)

Configure certificates for Apache, NGINX, Postfix, LDAP, etc.

Part 4: Automation (Chapters 22-26)

Master certmonger, crypto-policies, Let’s Encrypt, Ansible.

Part 5: Troubleshooting (Chapters 27-33) ⭐

This is where you become an expert! Systematic troubleshooting, common errors, emergency procedures.

Part 6: Migration (Chapters 34-37)

RHEL version upgrades and certificate migration.

Part 7: Security (Chapters 38-41)

FIPS mode, compliance, hardening, auditing.

Appendices

Optional advanced topics (Kubernetes, Vault, Zero Trust, etc.)

2.11 How to Use This Tutorial

For New Users

📖 Read chapters in order. Each builds on previous knowledge.

For Experienced Users

🎯 Jump to troubleshooting (Part 5) or specific services (Part 3).

For Support Engineers

🚨 Start with Chapter 27 (Troubleshooting Methodology), then dive into specifics.

Hands-On Labs

Every chapter includes practical examples. You’ll need:

• A RHEL system (VM or container is fine)
• Root or sudo access
• Internet connectivity (for package installs)

2.12 Key Concepts to Master

By the end of this tutorial, you’ll understand:

• ✅ What certificates are and why RHEL uses them
• ✅ How trust works on RHEL systems
• ✅ Where certificates live (/etc/pki/)
• ✅ Which tools to use (openssl, certutil, certmonger)
• ✅ Version differences (RHEL 7 vs 8 vs 9 vs 10)
• ✅ How to troubleshoot any certificate issue
• ✅ How to automate certificate lifecycle
• ✅ How to secure systems (FIPS, compliance)

2.13 Real-World Impact

Certificate issues cause:

• ❌ Service outages (expired certs)
• ❌ Security vulnerabilities (weak ciphers)
• ❌ Failed migrations (RHEL upgrades)
• ❌ Compliance failures (audit rejections)
• ❌ Lost productivity (troubleshooting time)

After this tutorial:

• ✅ Prevent issues before they happen
• ✅ Troubleshoot problems in minutes, not hours
• ✅ Automate certificate management
• ✅ Pass security audits
• ✅ Confidently migrate RHEL versions

2.14 Your First Exercise

Let’s verify your RHEL system is ready:

# Check RHEL version
cat /etc/redhat-release

# Check OpenSSL
openssl version

# Check if certmonger is installed
rpm -q certmonger

# Check if you can use sudo
sudo whoami

# Check internet connectivity (for package installs)
ping -c 3 access.redhat.com

# List current trusted CAs (sample)
trust list | head -20

✅ If all commands work, you’re ready to proceed!

2.15 Let’s Begin!

You now understand:

• What certificates are
• Why they matter on RHEL
• Where they live on the filesystem
• Which tools you’ll use
• What you’ll learn in this tutorial

Ready to dive deeper?

Quick Reference

┌────────────────────────────────────────────────────────────┐
│ CERTIFICATE QUICK START (RHEL)                             │
├────────────────────────────────────────────────────────────┤
│ View cert:     openssl x509 -in cert.crt -noout -text      │
│ Check expiry:  openssl x509 -in cert.crt -noout -dates     │
│ Add CA:        cp ca.crt /etc/pki/ca-trust/source/anchors/ │
│                sudo update-ca-trust                        │
│ List tracked:  getcert list                                │
│ Check policy:  update-crypto-policies --show  (RHEL 8+)    │
└────────────────────────────────────────────────────────────┘

Cert location:  /etc/pki/tls/certs/
Key location:   /etc/pki/tls/private/  (mode 600!)
CA trust:       /etc/pki/ca-trust/

🧪 Hands-On Lab

Lab 01: Environment Setup

Validate your RHEL environment and install essential certificate management tools

• 📁 Location: labs/en_US/01-environment-setup/
• ⏱️ Time: 15-20 minutes
• 🎯 Level: Beginner

Chapter Navigation

Chapter 3: RHEL Certificate Tools Overview

Learning Objective: Get familiar with the essential tools for managing certificates on RHEL so you know which tool to use for each task.

3.1 Your Certificate Toolkit

When working with certificates on RHEL, you’ll use these core tools:

3.2 OpenSSL - The Swiss Army Knife

Available: All RHEL versions Package: openssl

Version Differences

# Check your version
openssl version

# RHEL 7: OpenSSL 1.0.2k-26
# RHEL 8: OpenSSL 1.1.1k-14
# RHEL 9: OpenSSL 3.5.5-2
# RHEL 10: OpenSSL 3.5.5-2

Common Uses

#============================================#
# INSPECT CERTIFICATES
#============================================#

# View certificate details
openssl x509 -in cert.crt -noout -text

# Check expiration
openssl x509 -in cert.crt -noout -dates
openssl x509 -in cert.crt -noout -checkend 86400  # Check if expires in 24h

# View certificate subject
openssl x509 -in cert.crt -noout -subject -issuer


#============================================#
# GENERATE KEYS
#============================================#

# RHEL 7 style (still works on all versions)
openssl genrsa -out server.key 2048

# RHEL 8+ modern style (recommended)
openssl genpkey -algorithm RSA -out server.key -pkeyopt rsa_keygen_bits:2048

# RHEL 9+ EC key (elliptic curve)
openssl genpkey -algorithm EC -out ec.key -pkeyopt ec_paramgen_curve:P-256


#============================================#
# CREATE CSR (Certificate Signing Request)
#============================================#

# Basic CSR
openssl req -new -key server.key -out server.csr \
  -subj "/C=US/ST=State/L=City/O=Organization/CN=server.example.com"

# CSR with SANs (required for modern browsers!)
openssl req -new -key server.key -out server.csr \
  -subj "/CN=server.example.com" \
  -addext "subjectAltName=DNS:server.example.com,DNS:www.example.com"


#============================================#
# TEST CONNECTIONS
#============================================#

# Test HTTPS
openssl s_client -connect server.example.com:443 -servername server.example.com

# Test specific TLS version
openssl s_client -connect server.example.com:443 -tls1_2
openssl s_client -connect server.example.com:443 -tls1_3

# Test LDAPS
openssl s_client -connect ldap.example.com:636

# Test SMTP with STARTTLS
openssl s_client -connect mail.example.com:25 -starttls smtp

Version-Specific Differences

RHEL 7 (OpenSSL 1.0.2k):

• ✅ Stable and well-tested
• ❌ No TLS 1.3 support
• ❌ Older command syntax

RHEL 8 (OpenSSL 1.1.1k):

• ✅ TLS 1.3 support
• ✅ Modern command syntax
• ✅ Better defaults

RHEL 9/10 (OpenSSL 3.5.5):

• ✅ Provider architecture
• ✅ Enhanced FIPS support
• ⚠️ API changes (affects custom apps)
• ⚠️ Legacy algorithms require -provider legacy

3.3 certutil - NSS Database Tool

Available: All RHEL versions Package: nss-tools

Used for Mozilla/Firefox-style certificate databases.

Common Uses

#============================================#
# MANAGE NSS DATABASE
#============================================#

# Create new database
certutil -N -d /etc/pki/nssdb

# List certificates
certutil -L -d /etc/pki/nssdb

# Add CA certificate
certutil -A -n "My CA" -t "CT,C,C" -d /etc/pki/nssdb -i ca.crt

# Delete certificate
certutil -D -n "Certificate Name" -d /etc/pki/nssdb

# Export certificate
certutil -L -n "Certificate Name" -d /etc/pki/nssdb -a > exported.crt

When to Use certutil

• Managing Firefox/Thunderbird certificates
• Working with applications that use NSS (many Red Hat services)
• When you see .db files in /etc/pki/nssdb/

3.4 update-ca-trust - Trust Store Management

Available: All RHEL versions Package: ca-certificates (installed by default)

Manages which Certificate Authorities (CAs) your system trusts.

How It Works

Your Custom CAs
  ↓
/etc/pki/ca-trust/source/anchors/
  ↓
update-ca-trust extract
  ↓
/etc/pki/ca-trust/extracted/
  ├── pem/tls-ca-bundle.pem       (OpenSSL/Python/Ruby)
  ├── openssl/ca-bundle.trust.crt (OpenSSL specific)
  └── java/cacerts                (Java applications)

Common Uses

#============================================#
# ADD CUSTOM CA
#============================================#

# Step 1: Copy CA certificate
sudo cp corporate-ca.crt /etc/pki/ca-trust/source/anchors/

# Step 2: Update trust store
sudo update-ca-trust extract

# That's it! Now all applications trust this CA


#============================================#
# REMOVE/BLACKLIST CA (RHEL 8+)
#============================================#

# Blacklist a compromised CA
sudo cp compromised-ca.crt /etc/pki/ca-trust/source/blacklist/
sudo update-ca-trust extract


#============================================#
# VERIFY TRUST
#============================================#

# Check if certificate is trusted
openssl verify /path/to/cert.crt

# List all trusted CAs
trust list | grep "certificate-authority"

# Search for specific CA
trust list | grep -i "Let's Encrypt"

Key Directories

/etc/pki/ca-trust/
├── source/
│   ├── anchors/          ← Add your trusted CAs here
│   └── blacklist/        ← Blacklist CAs (RHEL 8+)
└── extracted/
    ├── pem/              ← Used by most apps
    ├── openssl/          ← OpenSSL-specific
    └── java/             ← Java applications

3.5 certmonger - Automatic Certificate Renewal

Available: All RHEL versions Package: certmonger

The “set it and forget it” tool for certificates.

What It Does

certmonger:

• Tracks certificate expiration dates
• Automatically renews before expiry
• Works with multiple CA workflows (IPA, local CA, external helpers)
• Runs post-renewal commands (e.g., restart services)

Basic Workflow

#============================================#
# INSTALLATION
#============================================#

sudo dnf install certmonger  # RHEL 8/9/10; use yum on RHEL 7
sudo systemctl enable --now certmonger


#============================================#
# REQUEST CERTIFICATE
#============================================#

# From FreeIPA
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/web.crt \
  -k /etc/pki/tls/private/web.key \
  -D web.example.com \
  -K host/web.example.com@REALM

# Self-signed (for testing)
sudo getcert request \
  -f /etc/pki/tls/certs/test.crt \
  -k /etc/pki/tls/private/test.key


#============================================#
# MONITOR CERTIFICATES
#============================================#

# List all tracked certificates
sudo getcert list

# Check specific certificate
sudo getcert list -f /etc/pki/tls/certs/web.crt

# Watch for renewal
sudo journalctl -u certmonger -f

Key Features by Version

RHEL 7:

• Basic tracking and renewal
• IPA integration
• Manual configuration

RHEL 8:

• Enhanced IPA integration
• Better error reporting
• Post-save commands

RHEL 9:

• Improved monitoring
• Better status reporting
• Same strong fit for IPA and tracked renewals

# RHEL 9 - Native FreeIPA integration
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/internal.crt \
  -k /etc/pki/tls/private/internal.key \
  -K HTTP/$(hostname -f)@REALM \
  -D $(hostname -f) \
  -C "systemctl reload httpd"

3.6 crypto-policies - System-Wide Security (RHEL 8+)

Available: RHEL 8, 9, 10 only Package: crypto-policies (installed by default)

GAME CHANGER: Control TLS versions, ciphers, and key sizes system-wide!

The Big Idea

Instead of configuring every application individually:

❌ OLD WAY (RHEL 7):
- Configure Apache SSL ciphers
- Configure NGINX SSL ciphers
- Configure Postfix TLS settings
- Configure OpenLDAP TLS settings
- Configure every application...

✅ NEW WAY (RHEL 8+):
- Set ONE system policy
- All applications follow it automatically!

Available Policies

# Check current policy
update-crypto-policies --show

# Policies:
# DEFAULT  - Balanced security (TLS 1.2+, RSA 2048+)
# LEGACY   - Compatibility mode (allows TLS 1.0/1.1)
# FUTURE   - Stricter security (TLS 1.2+, RSA 3072+)
# FIPS     - Federal compliance mode

Policy Comparison

Common Uses

#============================================#
# CHANGE POLICY
#============================================#

# Set FUTURE policy (stricter)
sudo update-crypto-policies --set FUTURE
# Reboot or restart services

# Temporarily use LEGACY (for old systems)
sudo update-crypto-policies --set LEGACY
# Note: LEGACY should be temporary!


#============================================#
# CUSTOM POLICIES (RHEL 9+)
#============================================#

# Subpolicies - modify existing policy
sudo update-crypto-policies --set DEFAULT:NO-SHA1
sudo update-crypto-policies --set FUTURE:AD-SUPPORT


#============================================#
# TROUBLESHOOT POLICY ISSUES
#============================================#

# If service fails after policy change:
# 1. Check current policy
update-crypto-policies --show

# 2. Check application config
cat /etc/crypto-policies/back-ends/opensslcnf.config

# 3. Test with LEGACY temporarily
sudo update-crypto-policies --set LEGACY
sudo systemctl restart <service>

What crypto-policies Controls

Automatically configures:

• OpenSSL
• GnuTLS
• NSS
• OpenJDK/Java
• BIND
• Kerberos
• OpenSSH
• And more!

Bottom line: Change one setting, update system-wide security. Brilliant!

3.7 Tool Selection Guide

“Which tool should I use?”

┌─────────────────────────────────────────────────────────────┐
│ CERTIFICATE TOOL DECISION TREE                              │
└─────────────────────────────────────────────────────────────┘

I need to...
│
├─ Inspect a certificate
│  └─ Use: openssl x509 -in cert.crt -noout -text
│
├─ Generate a key/CSR
│  └─ Use: openssl genpkey / openssl req
│
├─ Test a TLS connection
│  └─ Use: openssl s_client -connect host:port
│
├─ Add a trusted CA system-wide
│  └─ Use: copy to /etc/pki/ca-trust/source/anchors/
│          then: update-ca-trust
│
├─ Auto-renew certificates
│  └─ Use: certmonger (getcert/ipa-getcert)
│
├─ Change system TLS policy (RHEL 8+)
│  └─ Use: update-crypto-policies --set <POLICY>
│
├─ Work with Firefox/NSS databases
│  └─ Use: certutil
│
└─ Troubleshoot certificate issues
   └─ Use: Chapter 27 methodology!

3.8 Tool Availability Matrix

3.9 Installation Check

Verify you have the essential tools:

#============================================#
# CHECK INSTALLED TOOLS
#============================================#

# OpenSSL (should be installed by default)
openssl version

# NSS tools
rpm -q nss-tools || echo "Install with: sudo dnf install nss-tools"

# certmonger
rpm -q certmonger || echo "Install with: sudo dnf install certmonger (RHEL 8+); sudo yum install certmonger (RHEL 7)"

# Check crypto-policies (RHEL 8+ only)
which update-crypto-policies &>/dev/null && \
  echo "Crypto-policies available: $(update-crypto-policies --show)" || \
  echo "Crypto-policies not available (RHEL 7 or earlier)"

3.10 Quick Reference Commands

# === OpenSSL ===
openssl version                          # Check version
openssl x509 -in cert.crt -noout -text   # Inspect certificate
openssl s_client -connect host:443       # Test HTTPS

# === Trust Store ===
sudo cp ca.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust                     # Add trusted CA

# === certmonger ===
sudo getcert list                        # List tracked certs
sudo getcert list -f /path/to/cert.crt   # Check specific cert
sudo journalctl -u certmonger -f         # Watch logs

# === Crypto-Policies (RHEL 8+) ===
update-crypto-policies --show            # Current policy
sudo update-crypto-policies --set <POL>  # Change policy

# === NSS ===
certutil -L -d /etc/pki/nssdb            # List NSS certs

3.11 What’s Next?

Now that you know the tools, you’ll learn:

• Chapter 4: Basic cryptography concepts
• Chapter 5: Understanding X.509 certificates
• Chapter 6: RHEL trust store deep dive
• Chapter 22: certmonger mastery (detailed)
• Chapter 23: Crypto-policies deep dive (detailed)

Quick Reference Card

┌──────────────────────────────────────────────────────────┐
│ RHEL CERTIFICATE TOOLS CHEAT SHEET                       │
├──────────────────────────────────────────────────────────┤
│ Inspect:     openssl x509 -in cert.crt -noout -text      │
│ Test:        openssl s_client -connect host:443          │
│ Add CA:      cp ca.crt /etc/pki/ca-trust/source/anchors/ │
│              sudo update-ca-trust                        │
│ Auto-renew:  sudo getcert list                           │
│ Policy:      update-crypto-policies --show  (RHEL 8+)    │
│ NSS:         certutil -L -d /etc/pki/nssdb               │
└──────────────────────────────────────────────────────────┘

Chapter Navigation

Chapter 4: Basic Cryptography for RHEL Admins

Practical Focus: Learn the cryptography concepts you need to manage certificates on RHEL - no PhD required!

4.1 Symmetric vs Asymmetric

Symmetric crypto (e.g., AES) relies on a single shared secret. In contrast, asymmetric crypto provides two complementary keys:

• Public key — share freely, used for encryption or signature verification.
• Private key — keep secret, used for decryption or signing.

4.2 RSA in a Nutshell

1. Select two large primes p and q.
2. Compute modulus n = p × q.
3. Derive public exponent e and private exponent d such that e × d ≡ 1 (mod φ(n)).
4. The pair (n, e) is public; (n, d) is private.

Strength derives from the difficulty of factoring n.

4.3 Elliptic-Curve Cryptography (ECC)

ECC offers comparable security with much smaller key sizes by operating over points on elliptic curves. Popular curves include secp256r1 (P-256) and Curve25519.

4.4 Key Generation Lab (OpenSSL)

# Generate 3072-bit RSA private key
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:3072 -out rsa.key.pem

# Extract public key
openssl pkey -in rsa.key.pem -pubout -out rsa.pub.pem

# Generate P-256 EC key-pair
openssl ecparam -genkey -name prime256v1 -out ec.key.pem
openssl pkey -in ec.key.pem -pubout -out ec.pub.pem

We will reuse these keys in later chapters to create certificates.

4.5 Hybrid Encryption

In TLS a symmetric session key is exchanged using asymmetric crypto (RSA or ECDHE). This yields the best of both worlds: efficiency and secure key exchange.

4.6 RHEL-Specific Considerations

Key Generation on RHEL by Version

RHEL 7 (OpenSSL 1.0.2k):

# Old style (still works on all versions)
openssl genrsa -out server.key 2048

# Extract public key
openssl rsa -in server.key -pubout -out server.pub

RHEL 8+ (OpenSSL 1.1.1k / 3.5.5):

# Modern style (recommended)
openssl genpkey -algorithm RSA -out server.key -pkeyopt rsa_keygen_bits:2048

# Extract public key
openssl pkey -in server.key -pubout -out server.pub

Minimum Key Sizes by RHEL Version

Recommendation: Always use RSA 2048+ or ECC P-256+ for compatibility!

Testing on RHEL

# Generate test key pair (RHEL 8+)
openssl genpkey -algorithm RSA -out test.key -pkeyopt rsa_keygen_bits:2048

# Verify key
openssl pkey -in test.key -text -noout

# Create test data
echo "Hello RHEL" > message.txt

# Sign with private key
openssl dgst -sha256 -sign test.key -out message.sig message.txt

# Verify with public key
openssl dgst -sha256 -verify test.pub -signature message.sig message.txt
# Verified OK

Quick Reference

┌─────────────────────────────────────────────────────────┐
│ CRYPTOGRAPHY FOR RHEL ADMINS                            │
├─────────────────────────────────────────────────────────┤
│ Asymmetric:   Public key (share) + Private key (secret) │
│ Algorithms:   RSA, ECC (Elliptic Curve)                 │
│                                                         │
│ RSA Sizes:    2048 bits (minimum on RHEL 8+)            │
│               4096 bits (recommended)                   │
│                                                         │
│ ECC Curves:   P-256 (secp256r1) - minimum               │
│               P-384 (secp384r1) - recommended           │
│                                                         │
│ RHEL 7:       openssl genrsa -out key 2048              │
│ RHEL 8/9/10:  openssl genpkey -algorithm RSA -out key   │
│                                                         │
│ Use case:     TLS/SSL certificates                      │
│ Security:     Private key MUST be protected (chmod 600) │
└─────────────────────────────────────────────────────────┘

🧪 Hands-On Lab

Lab 02: Key Generation

Practice generating cryptographic key pairs in this hands-on exercise:

• Generate RSA keys (2048-bit and 4096-bit)

• Generate ECC keys (P-256 and P-384)

• Extract public keys from private keys

• Compare key sizes and performance

• 📁 Location: labs/en_US/02-key-generation/

• ⏱️ Time: 20-25 minutes

• 🎯 Level: Beginner

Chapter Navigation

Chapter 5: X.509 Certificates on RHEL

Standard Format: X.509 is the certificate standard used everywhere on RHEL. Learn its structure and how to work with it on Red Hat systems.

5.1 Origins of the Standard

X.509 emerged from the X.500 directory project (ITU-T, 1988) to define a standard identity certificate—a document that binds a public key to a subject name, signed by a trusted authority.

5.2 Certificate Anatomy

5.3 Common Extensions

• Subject Alternative Name (SAN) — Hosts/IPs bound to cert.
• Key Usage / Extended Key Usage — Permitted operations (TLS server, code signing…).
• Basic Constraints — Indicates if cert can sign others (CA:TRUE).

5.4 Viewing a Certificate

openssl x509 -in server.crt -noout -text

Observe each section matches the table above.

5.5 PEM vs DER Encodings

• PEM — Base64 + -----BEGIN CERTIFICATE----- headers (most common on RHEL).
• DER — Binary ASN.1, useful for embedded devices.

5.6 X.509 on RHEL Systems

Certificate Locations on RHEL

# Standard RHEL certificate locations
/etc/pki/tls/certs/          # Server certificates (public)
/etc/pki/tls/private/        # Private keys (mode 600!)
/etc/pki/ca-trust/           # Trusted CA certificates
/etc/pki/nssdb/              # NSS database (Firefox, etc.)

# Service-specific locations
/etc/httpd/conf/ssl.crt/     # Apache (alternative)
/etc/nginx/certs/            # NGINX (custom)
/var/lib/pgsql/data/         # PostgreSQL
/etc/openldap/certs/         # OpenLDAP

Viewing Certificates on RHEL

# View full certificate details
openssl x509 -in /etc/pki/tls/certs/server.crt -noout -text

# Quick checks (RHEL sysadmin focus)
openssl x509 -in server.crt -noout -subject             # Who is it for?
openssl x509 -in server.crt -noout -issuer              # Who signed it?
openssl x509 -in server.crt -noout -dates               # When is it valid?
openssl x509 -in server.crt -noout -ext subjectAltName  # SANs (critical!)

# Check if expired
openssl x509 -in server.crt -noout -checkend 0
# Exit 0 = valid, Exit 1 = expired

RHEL Version Differences for X.509

Key Point: Modern browsers and RHEL 9+ require SANs (Subject Alternative Names)!

Creating X.509 Certificates on RHEL

# Complete workflow on RHEL

# Step 1: Generate private key
openssl genpkey -algorithm RSA -out server.key -pkeyopt rsa_keygen_bits:2048

# Step 2: Create CSR (Certificate Signing Request)
openssl req -new -key server.key -out server.csr \
  -subj "/C=US/ST=State/O=Company/CN=server.example.com" \
  -addext "subjectAltName=DNS:server.example.com,DNS:www.example.com"

# Step 3: Self-signed (testing only!)
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

# Step 4: View your X.509 certificate
openssl x509 -in server.crt -noout -text

# Step 5: Install on RHEL
sudo cp server.crt /etc/pki/tls/certs/
sudo cp server.key /etc/pki/tls/private/
sudo chmod 600 /etc/pki/tls/private/server.key

Quick Reference

┌─────────────────────────────────────────────────────────────────┐
│ X.509 CERTIFICATES ON RHEL                                      │
├─────────────────────────────────────────────────────────────────┤
│ Standard:  X.509 v3 (with extensions)                           │
│ Encoding:  PEM (Base64, human-readable)                         │
│                                                                 │
│ View:      openssl x509 -in cert.crt -noout -text               │
│ Subject:   openssl x509 -in cert.crt -noout -subject            │
│ Expiry:    openssl x509 -in cert.crt -noout -dates              │
│ SANs:      openssl x509 -in cert.crt -noout -ext subjectAltName │
│                                                                 │
│ Location:  /etc/pki/tls/certs/ (certificates)                   │
│            /etc/pki/tls/private/ (keys, mode 600!)              │
│                                                                 │
│ Critical:  SANs are REQUIRED on RHEL 9+                         │
│            SHA-256+ signature required on RHEL 8+               │
└─────────────────────────────────────────────────────────────────┘

🧪 Hands-On Lab

Lab 04: X.509 Certificates

Create self-signed certificates, generate CSRs, inspect certificates, and convert formats

• 📁 Location: labs/en_US/04-x509-certificates/
• ⏱️ Time: 25-30 minutes
• 🎯 Level: Beginner

Chapter Navigation

Chapter 6: RHEL Trust Store Deep Dive

Trust Architecture: Understanding how RHEL validates certificates and manages trusted CAs is essential for troubleshooting certificate problems. This chapter goes beyond the basics: it covers the internal mechanics of update-ca-trust, how p11-kit processes trust sources, what happens with duplicate certificates, and how to debug trust issues with trust list.

6.1 RHEL Trust Store Architecture

How RHEL Validates Certificates

When any application on RHEL validates a certificate:

1. Check certificate signature using issuer’s public key
2. Find issuer certificate in trust store
3. Repeat until reaching trusted root CA
4. Verify root CA is trusted by RHEL system

Trust store location: /etc/pki/ca-trust/

The Role of p11-kit

RHEL’s trust store is not a flat file that applications read directly. It is a managed system built on top of p11-kit, which provides a PKCS#11 trust module. The key components are:

Applications like curl, wget, OpenSSL, GnuTLS, and NSS all consume the extracted bundles. They never read the source directories directly.

6.2 Trust Store Directory Structure

RHEL 7/8:

/etc/pki/ca-trust/
├── source/
│   ├── anchors/                   ← Admin-added trusted CA certificates
│   ├── blacklist/                 ← Admin-added distrusted certificates
│   └── ca-bundle.legacy.crt       ← Legacy bundle (compatibility only)
├── extracted/
│   ├── pem/
│   │   ├── tls-ca-bundle.pem      ← For TLS clients (curl, wget, Python...)
│   │   ├── email-ca-bundle.pem    ← For S/MIME email validation
│   │   └── objsign-ca-bundle.pem  ← For code signing verification
│   ├── openssl/
│   │   └── ca-bundle.trust.crt    ← OpenSSL "trusted certificate" format
│   ├── java/
│   │   └── cacerts                ← Java JKS keystore
│   └── edk2/
│       └── cacerts.bin            ← UEFI firmware format
└── README

/usr/share/pki/ca-trust-source/
├── anchors/                       ← Package-provided trust anchors (from RPMs)
├── blacklist/                     ← Package-provided distrusted certificates
└── ca-bundle.trust.p11-kit        ← Mozilla CA bundle shipped by ca-certificates RPM

RHEL 9/10+: The blacklist/ directory was renamed to blocklist/:

/etc/pki/ca-trust/
├── source/
│   ├── anchors/                   ← Admin-added trusted CA certificates
│   ├── blocklist/                 ← Admin-added distrusted certificates
│   └── ca-bundle.legacy.crt       ← Legacy bundle (compatibility only)
├── extracted/
│   └── (same structure as above)
└── README

/usr/share/pki/ca-trust-source/
├── anchors/                       ← Package-provided trust anchors (from RPMs)
├── blocklist/                     ← Package-provided distrusted certificates
└── ca-bundle.trust.p11-kit        ← Mozilla CA bundle shipped by ca-certificates RPM

Naming convention: Throughout this chapter, blacklist/ refers to the RHEL 7/8 directory name and blocklist/ refers to the RHEL 9/10+ directory name. Both serve the same purpose. When you see a path like source/blacklist/, substitute source/blocklist/ on RHEL 9+.

Source Directories Priority

The update-ca-trust pipeline reads certificates from multiple source directories, processed in a defined order:

Within each location:

• anchors/ — Certificates placed here are treated as trusted CAs
• blacklist/ (RHEL 7/8) or blocklist/ (RHEL 9+) — Certificates placed here are treated as explicitly distrusted

The /etc/pki/ paths always override /usr/share/pki/ paths. This follows the standard RHEL convention: /usr/share/ holds package defaults, /etc/ holds admin customizations.

6.3 What Happens When You Run update-ca-trust

The Execution Pipeline

update-ca-trust is a shell script (inspect it yourself: cat /usr/bin/update-ca-trust). When you execute sudo update-ca-trust, the following sequence occurs:

Step 1: Collect all trust sources

p11-kit reads every certificate file from these directories:

/usr/share/pki/ca-trust-source/anchors/
/usr/share/pki/ca-trust-source/blacklist/    ← RHEL 7/8
/usr/share/pki/ca-trust-source/blocklist/    ← RHEL 9+
/usr/share/pki/ca-trust-source/ca-bundle.trust.p11-kit
/etc/pki/ca-trust/source/anchors/
/etc/pki/ca-trust/source/blacklist/          ← RHEL 7/8
/etc/pki/ca-trust/source/blocklist/          ← RHEL 9+

It accepts PEM (.pem, .crt), DER (.der), and p11-kit trust objects (.p11-kit) formats.

Step 2: Parse trust attributes

For each certificate, p11-kit determines its trust disposition. The PEM format of the certificate file matters:

• -----BEGIN TRUSTED CERTIFICATE----- (OpenSSL “trusted” format) — Contains the certificate plus auxiliary trust data: explicit lists of trusted and rejected key usage OIDs. p11-kit reads these embedded trust/reject attributes directly.
• -----BEGIN CERTIFICATE----- (plain PEM) or DER — Contains only the certificate with no trust metadata. p11-kit assigns trust based solely on the directory where the file is placed (anchors/ = trusted, blacklist//blocklist/ = distrusted).
• .p11-kit format files — Contain PKCS#11 trust objects with fine-grained attributes (trusted, x-distrusted, purpose OIDs). Used by the ca-bundle.trust.p11-kit file shipped with the ca-certificates package.

Critical: The BEGIN TRUSTED CERTIFICATE and BEGIN CERTIFICATE formats are not interchangeable. A BEGIN TRUSTED CERTIFICATE file carries auxiliary trust data — explicit lists of trusted and/or rejected use OIDs. When no uses are listed (empty trust attributes), p11-kit interprets this as “trusted for nothing” — effectively distrusted. If the same certificate also exists as a plain BEGIN CERTIFICATE (which implies trust for all purposes), p11-kit detects contradictory trust assertions and marks the certificate as distrusted. This conflict occurs in two cases: (1) the BEGIN TRUSTED CERTIFICATE has explicit rejected uses, or (2) it has empty trust attributes (no trusted uses, no rejected uses). See Section 6.6 for details.

Step 3: Merge and resolve conflicts

When the same certificate (matched by its DER-encoded content) appears in multiple source locations, p11-kit applies merge rules:

1. Distrust wins over trust. If a certificate appears in both anchors/ and blacklist//blocklist/, it is distrusted.

2. Admin overrides packages. Trust attributes set in /etc/pki/ca-trust/source/ override those from /usr/share/pki/ca-trust-source/.

3. Explicit attributes override defaults. A .p11-kit file with specific purpose restrictions overrides the blanket trust given to a plain PEM in anchors/.

4. Conflicting trust formats cause distrust. If the same certificate (identical fingerprint) appears as both BEGIN TRUSTED CERTIFICATE (or .p11-kit) and BEGIN CERTIFICATE, distrust occurs in two cases:

   • The BEGIN TRUSTED CERTIFICATE has explicit rejected uses — the rejection contradicts the plain PEM’s implicit full trust.
   • The BEGIN TRUSTED CERTIFICATE has empty trust attributes (no trusted uses, no rejected uses) — p11-kit interprets “no uses listed” as “trusted for nothing,” which contradicts the plain PEM’s implicit “trusted for all purposes.”

   In both cases, p11-kit marks the certificate as distrusted.

This is the single most common cause of unexpected distrust. An administrator copies a plain PEM file into source/anchors/ not realizing that the same certificate already exists in ca-bundle.trust.p11-kit with rejected use attributes or empty trust. The result: the CA is distrusted after update-ca-trust, and services that depend on it break silently.

Step 4: Extract to format-specific bundles

p11-kit runs extraction commands for each output format:

# PEM bundle for TLS (most commonly consumed)
p11-kit extract --format=pem-bundle \
    --filter=ca-anchors \
    --overwrite \
    --purpose=server-auth \
    /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

# PEM bundle for email (S/MIME)
p11-kit extract --format=pem-bundle \
    --filter=ca-anchors \
    --overwrite \
    --purpose=email-protection \
    /etc/pki/ca-trust/extracted/pem/email-ca-bundle.pem

# PEM bundle for code signing
p11-kit extract --format=pem-bundle \
    --filter=ca-anchors \
    --overwrite \
    --purpose=code-signing \
    /etc/pki/ca-trust/extracted/pem/objsign-ca-bundle.pem

# OpenSSL "trusted certificate" format (includes trust/reject attributes)
p11-kit extract --format=openssl-bundle \
    --filter=certificates \
    --overwrite \
    /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt

# Java keystore
p11-kit extract --format=java-cacerts \
    --filter=ca-anchors \
    --overwrite \
    --purpose=server-auth \
    /etc/pki/ca-trust/extracted/java/cacerts

# UEFI EDK2 format
p11-kit extract --format=edk2-cacerts \
    --filter=ca-anchors \
    --overwrite \
    --purpose=server-auth \
    /etc/pki/ca-trust/extracted/edk2/cacerts.bin

Step 5: Update compatibility symlinks

The system maintains symlinks so that legacy paths point to the extracted bundles:

/etc/pki/tls/certs/ca-bundle.crt → /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
/etc/pki/tls/certs/ca-bundle.trust.crt → /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
/etc/pki/java/cacerts → /etc/pki/ca-trust/extracted/java/cacerts

update-ca-trust vs update-ca-trust extract

These are functionally identical. The update-ca-trust script accepts extract as a subcommand, but running update-ca-trust with no arguments defaults to the extract action. There is no difference in behavior.

# These two commands produce identical results:
sudo update-ca-trust
sudo update-ca-trust extract

The extract subcommand exists for explicitness in scripts and documentation. Historically, update-ca-trust also supported enable and disable subcommands (to switch between the new p11-kit-managed trust store and the legacy flat-file approach), but those are no longer relevant on modern RHEL systems where p11-kit management is always active.

# Check current status (informational only on modern RHEL)
update-ca-trust check

Verifying What Changed

After running update-ca-trust, you can verify the result:

# Count trusted CAs in the PEM bundle
grep -c '^-----BEGIN CERTIFICATE-----' /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

# List all trusted CAs via p11-kit
trust list --filter=ca-anchors | grep "label:" | wc -l

# Check if a specific CA is present
trust list | grep -A 4 "My Company CA"

6.4 Adding Custom CA Certificates

Step-by-Step

# Step 1: Copy CA certificate (PEM or DER format)
sudo cp company-ca.crt /etc/pki/ca-trust/source/anchors/

# Step 2: Update trust store
sudo update-ca-trust

# Step 3: Verify addition
trust list | grep -i "company"

# Step 4: Test
curl https://internal-server.example.com

Works identically on RHEL 7, 8, 9, 10.

Adding with Purpose Restrictions

If a CA should only be trusted for TLS server authentication (not email signing or code signing), use the trust command instead of the simple file copy approach:

# Trust only for TLS server authentication
sudo trust anchor --store /path/to/company-ca.crt

# Or with explicit purpose restriction via p11-kit format
# Create a .p11-kit file with restricted trust:
cat > /tmp/company-ca.p11-kit <<'EOF'
[p11-kit-object-v1]
class: x-certificate-extension
label: "My Company CA"
x-public-key-info: <extracted-from-cert>

[p11-kit-object-v1]
class: certificate
label: "My Company CA"
certificate-type: x-509
java-midp-security-domain: 0
trusted: true
x-distrusted: false

[p11-kit-object-v1]
class: x-certificate-extension
label: "My Company CA"
object-id: 2.5.29.37
value: "%06%08%2b%06%01%05%05%07%03%01"
EOF

The trust anchor --store command handles this automatically and is the recommended approach on RHEL 8+.

6.5 RHEL Version Trust Store Features

Trust Management Evolution

RHEL 8+ Enhancement:

# Advanced trust management (RHEL 8+)
trust anchor /path/to/ca.crt --purpose server-auth
trust anchor --remove "pkcs11:id=%CERT_ID%"
trust list --filter=ca-anchors

# Distrust a compromised CA
# RHEL 7/8:
sudo cp compromised.crt /etc/pki/ca-trust/source/blacklist/
# RHEL 9+:
sudo cp compromised.crt /etc/pki/ca-trust/source/blocklist/

sudo update-ca-trust

6.6 Duplicate Certificates: What Happens and How to Handle Them

PEM Format Variants and Trust Implications

Before discussing duplicates, it is essential to understand the three PEM header formats RHEL uses, because the format itself carries trust semantics:

You can inspect the trust attributes embedded in a BEGIN TRUSTED CERTIFICATE file:

# Show the auxiliary trust data
openssl x509 -in cert.crt -noout -text -trustout 2>/dev/null | grep -A 5 "Trusted Uses\|Rejected Uses"

A BEGIN TRUSTED CERTIFICATE file with “Rejected Uses: TLS Web Server Authentication” and a BEGIN CERTIFICATE file for the same cert (which carries no rejection information) are contradictory from p11-kit’s perspective. The same applies to a BEGIN TRUSTED CERTIFICATE with no trust/reject uses at all — p11-kit interprets empty trust as “trusted for nothing,” which contradicts the implicit full trust of a plain BEGIN CERTIFICATE.

Understanding “Duplicate” Certificates

Two certificates can be duplicates at different levels:

The critical distinction: p11-kit matches duplicates by DER-encoded certificate content, not by metadata fields. However, “matching” the DER content is only half the story — what happens next depends entirely on whether the trust wrapping formats agree:

• If the raw certificate DER bytes are identical and all sources agree on trust disposition → deduplicated, trusted
• If the raw certificate DER bytes are identical but one source has rejected use OIDs embedded and the other does not → contradictory trust assertions → DISTRUSTED
• If the raw certificate DER bytes are identical but one source has BEGIN TRUSTED CERTIFICATE with empty trust attributes (no trusted uses, no rejected uses) → p11-kit interprets this as “trusted for nothing” → contradicts implicit full trust → DISTRUSTED
• If the raw certificate DER bytes are identical and one source has BEGIN TRUSTED CERTIFICATE with explicit trusted uses only (no rejected uses) → certificate is accepted with the specified trust purposes
• If the raw certificate DER bytes differ (even by a single byte) → p11-kit treats them as separate certificates and both are included

Scenario: Conflicting Trust Formats (Most Dangerous)

This is the single most common and most damaging “duplicate” problem. It happens silently when an administrator copies a certificate into source/anchors/ not realizing the same certificate already exists in the system bundle with rejected use OIDs or empty trust attributes in its trust data.

Example:

/usr/share/pki/ca-trust-source/ca-bundle.trust.p11-kit
  → Contains "My Corp CA" as a p11-kit trust object with specific trust attributes
    INCLUDING rejected uses (e.g., "Rejected Uses: TLS Web Server Authentication")
    OR with empty trust attributes (no trusted uses, no rejected uses listed)

/etc/pki/ca-trust/source/anchors/my-corp-ca.crt
  → Same "My Corp CA" as plain PEM (-----BEGIN CERTIFICATE-----)
    (no trust attributes — relies on directory placement for implicit trust)

Result: The same certificate DER content now has contradictory trust descriptions. The p11-kit trust object either explicitly rejects certain uses, or lists no uses at all (which p11-kit interprets as “trusted for nothing”). The plain PEM carries no trust metadata, implying trust for all purposes. p11-kit cannot reconcile these contradictions, and marks the certificate as distrusted. After running update-ca-trust:

• The certificate disappears from tls-ca-bundle.pem
• trust list shows it with trust: distrusted
• All services relying on this CA start failing with certificate verify failed

This same problem occurs with the OpenSSL BEGIN TRUSTED CERTIFICATE format when it contains rejected uses OR empty trust:

/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
  → Contains the certificate as -----BEGIN TRUSTED CERTIFICATE-----
    (with "Rejected Uses" OIDs embedded, OR with no trust/reject uses at all)

/etc/pki/ca-trust/source/anchors/certificate.crt
  → Same certificate as -----BEGIN CERTIFICATE-----
    (no embedded trust data — no rejection information)

Result: Same outcome — the rejection or empty trust in one source contradicts the implicit full trust in the other, certificate marked as distrusted.

Key insight: In a BEGIN TRUSTED CERTIFICATE, “no trust/reject uses listed” does NOT mean “trusted for all purposes.” It means “trusted for nothing.” This is the critical distinction from a plain BEGIN CERTIFICATE in anchors/, where directory placement grants implicit full trust. To debug, search for duplicate certificates across /etc/pki/ca-trust/source/ and /usr/share/pki/ca-trust-source/ and check if any BEGIN TRUSTED CERTIFICATE exists with no Reject or empty trust that would conflict with a plain PEM copy.

How to fix it:

# Option 1: Remove the plain PEM duplicate from anchors/
sudo rm /etc/pki/ca-trust/source/anchors/my-corp-ca.crt
sudo update-ca-trust

# Option 2: If you NEED the cert in anchors/, convert it to the
# trusted format matching the existing trust attributes:
openssl x509 -in my-corp-ca.crt -addtrust serverAuth \
    -addtrust emailProtection -out my-corp-ca-trusted.crt
sudo cp my-corp-ca-trusted.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

# Verify the fix
trust list --filter=ca-anchors | grep -A 3 "My Corp CA"

Scenario: Explicit Distrusting

/usr/share/pki/ca-trust-source/ca-bundle.trust.p11-kit
  → Contains "Legacy Corp CA" with full trust

/etc/pki/ca-trust/source/blacklist/legacy-corp-ca.crt    ← RHEL 7/8
/etc/pki/ca-trust/source/blocklist/legacy-corp-ca.crt    ← RHEL 9+
  → Same "Legacy Corp CA" as plain PEM, placed in distrust directory

Result: Distrust wins by design. The certificate is excluded from tls-ca-bundle.pem and marked as distrusted in the OpenSSL bundle. This is the expected behavior when an admin intentionally distrusts a CA.

Scenario: Nearly Identical Certificates (Different DER)

A subtler problem occurs when two certificates look the same but are not byte-for-byte identical:

• A CA certificate was downloaded from two different sources with slightly different PEM line wrapping
• A CA certificate was re-encoded (PEM → DER → PEM) and gained different header metadata
• A CA was reissued with the same Subject DN but a new key pair and serial number

In these cases, p11-kit treats them as distinct certificates, and both end up in the extracted bundles. This can cause:

• Confusing trust list output with apparent duplicates
• Increased bundle size (cosmetic issue)
• Chain building confusion if one copy is distrusted and the other is trusted but they have different DER content

How to Detect Duplicate Certificates

Method 1: Fingerprint + format comparison

Extract fingerprints from all source certificates and check for duplicates, including what format each file uses:

# Scan all source directories for duplicate SHA-256 fingerprints
# AND report the PEM format of each file
for dir in /usr/share/pki/ca-trust-source/anchors \
           /etc/pki/ca-trust/source/anchors; do
    for cert in "$dir"/*.crt "$dir"/*.pem 2>/dev/null; do
        [ -f "$cert" ] || continue
        fp=$(openssl x509 -in "$cert" -noout -fingerprint -sha256 2>/dev/null)
        fmt=$(head -1 "$cert" 2>/dev/null)
        [ -n "$fp" ] && echo "$fp  [$fmt]  $cert"
    done
done | sort

# Look for same fingerprint appearing with different formats:
# If you see the same fingerprint with both "BEGIN CERTIFICATE" and
# "BEGIN TRUSTED CERTIFICATE", that is the source of a trust conflict.

If the same fingerprint appears with different PEM headers, you have found a trust format conflict that will cause distrust.

Method 2: Using trust list to spot duplicates

# Extract all labels and look for duplicates
trust list --filter=ca-anchors | grep "^    label:" | sort | uniq -c | sort -rn | head -20

If any label appears more than once, investigate further:

# Show full details for a suspected duplicate
trust list | grep -B 2 -A 10 "label: DigiCert Global Root G2"

Method 3: Compare the extracted bundle against source files

# Count certificates in the PEM bundle
grep -c 'BEGIN CERTIFICATE' /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

# Count unique certificates by fingerprint
awk '/BEGIN CERT/,/END CERT/' /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem | \
    csplit -z -f /tmp/cert- - '/BEGIN CERTIFICATE/' '{*}' 2>/dev/null
for f in /tmp/cert-*; do
    openssl x509 -in "$f" -noout -fingerprint -sha256 2>/dev/null
done | sort -u | wc -l
rm -f /tmp/cert-*

If the certificate count exceeds the unique fingerprint count, duplicates exist in the extracted bundle.

How to Identify Differences Between Suspected Duplicates

When you have two certificate files that appear to be the same CA, compare them at four levels:

Level 1: Check the PEM format (most common cause of conflicts)

# Check what PEM header each file uses
head -1 cert1.crt
head -1 cert2.crt

# "-----BEGIN CERTIFICATE-----"         → plain PEM, no trust data
# "-----BEGIN TRUSTED CERTIFICATE-----" → OpenSSL trusted format, HAS trust data

If one says BEGIN CERTIFICATE and the other says BEGIN TRUSTED CERTIFICATE, you have found the problem. They will cause a trust conflict even if the underlying certificate is identical.

Level 2: Compare DER-encoded certificate content

# Compare DER-encoded content (strips PEM headers, whitespace differences)
openssl x509 -in cert1.crt -outform DER -out /tmp/cert1.der
openssl x509 -in cert2.crt -outform DER -out /tmp/cert2.der
diff /tmp/cert1.der /tmp/cert2.der && echo "IDENTICAL DER" || echo "DIFFERENT DER"

Level 3: Compare certificate fields

# Compare subject, issuer, serial, validity, and key
for field in subject issuer serial dates fingerprint pubkey; do
    echo "=== $field ==="
    echo "cert1: $(openssl x509 -in cert1.crt -noout -$field 2>/dev/null)"
    echo "cert2: $(openssl x509 -in cert2.crt -noout -$field 2>/dev/null)"
done

Level 4: Compare embedded trust attributes (if TRUSTED CERTIFICATE format)

# Show trust attributes for each file
echo "=== cert1 trust attributes ==="
openssl x509 -in cert1.crt -noout -text -trustout 2>/dev/null | grep -A 5 "Trusted Uses\|Rejected Uses"
echo "=== cert2 trust attributes ==="
openssl x509 -in cert2.crt -noout -text -trustout 2>/dev/null | grep -A 5 "Trusted Uses\|Rejected Uses"

Common findings:

Cleaning Up Duplicates

If the certificate is already in the system bundle (most common case):

A certificate in anchors/ that already exists in ca-bundle.trust.p11-kit is not always harmless — if the system bundle copy has rejected use OIDs or empty trust attributes, the plain PEM duplicate causes distrust. Always check the trust attributes before assuming a duplicate is benign.

# 1. Check the format of the file in anchors/
head -1 /etc/pki/ca-trust/source/anchors/suspect.crt

# 2. Find the fingerprint
openssl x509 -in /etc/pki/ca-trust/source/anchors/suspect.crt -noout -fingerprint -sha256

# 3. Check if that certificate exists in the system bundle
trust list | grep -B5 -A5 "<subject from the cert>"

# 4. If the certificate already exists in the system bundle with proper
#    trust attributes, remove the duplicate from anchors/:
sudo rm /etc/pki/ca-trust/source/anchors/suspect.crt
sudo update-ca-trust

# 5. Verify the certificate is now trusted (not distrusted)
trust list --filter=ca-anchors | grep -A 3 "<subject from the cert>"

If you need to keep the certificate in anchors/ (custom CA not in system bundle):

# Ensure the file format matches what p11-kit expects.
# For custom CAs not in the system bundle, plain PEM is fine:
openssl x509 -in suspect.crt -out /etc/pki/ca-trust/source/anchors/suspect.crt
sudo update-ca-trust

6.7 Using trust list to Identify Distrusted Certificates

The trust list command is the primary tool for inspecting the state of the trust store after update-ca-trust has been run.

Basic Usage

# List all trust objects (trusted + distrusted)
trust list

# List only trusted CA anchors
trust list --filter=ca-anchors

# List only distrusted certificates
trust list --filter=blacklist     # RHEL 7/8
trust list --filter=blocklist     # RHEL 9+

# List all certificates (without filtering by trust disposition)
trust list --filter=certificates

Anatomy of trust list Output

Each entry in trust list output looks like this:

pkcs11:id=%DE%28%F4%A4%FF%E5%B9%2F%A3%C5%03%D1%A3%49%A7%F9%96%2A%82%12;type=cert
    type: certificate
    label: DigiCert Global Root G2
    trust: anchor
    category: authority

pkcs11:id=%01%02%03...;type=cert
    type: certificate
    label: Legacy Compromised CA
    trust: distrusted
    category: authority

Finding Distrusted Certificates

# List all distrusted certificates with their labels
trust list --filter=blacklist     # RHEL 7/8
trust list --filter=blocklist     # RHEL 9+

# Count distrusted certificates
trust list --filter=blocklist | grep "^pkcs11:" | wc -l    # RHEL 9+

# Search for a specific distrusted certificate
trust list --filter=blocklist | grep -B 1 -A 4 "Symantec"  # RHEL 9+

Tracing a Distrusted Certificate to Its Source

When trust list --filter=blacklist (RHEL 7/8) or trust list --filter=blocklist (RHEL 9+) shows a certificate you did not expect to be distrusted, you need to find where the distrust originates:

# Step 1: Get the label of the distrusted certificate
trust list --filter=blacklist     # RHEL 7/8
trust list --filter=blocklist     # RHEL 9+
# Example output:
# pkcs11:id=%AB%CD...;type=cert
#     type: certificate
#     label: Suspicious CA
#     trust: distrusted
#     category: authority

# Step 2: Check admin distrust directory
# RHEL 7/8: blacklist/  |  RHEL 9+: blocklist/
for distrust_dir in /etc/pki/ca-trust/source/blacklist \
                    /etc/pki/ca-trust/source/blocklist; do
    [ -d "$distrust_dir" ] || continue
    echo "=== $distrust_dir ==="
    ls -la "$distrust_dir"/
    for f in "$distrust_dir"/*; do
        [ -f "$f" ] || continue
        subj=$(openssl x509 -in "$f" -noout -subject 2>/dev/null)
        echo "$f: $subj"
    done
done

# Step 3: Check package-provided distrust directory
for distrust_dir in /usr/share/pki/ca-trust-source/blacklist \
                    /usr/share/pki/ca-trust-source/blocklist; do
    [ -d "$distrust_dir" ] || continue
    echo "=== $distrust_dir ==="
    ls -la "$distrust_dir"/
    for f in "$distrust_dir"/*; do
        [ -f "$f" ] || continue
        subj=$(openssl x509 -in "$f" -noout -subject 2>/dev/null)
        echo "$f: $subj"
    done
done

# Step 4: Check the main p11-kit bundle for inline distrust attributes
grep -A 5 "Suspicious CA" /usr/share/pki/ca-trust-source/ca-bundle.trust.p11-kit
# Look for "x-distrusted: true" or "nss-mozilla-ca-policy: false"

Interpretation of results:

Practical Example: Debugging a Distrusted CA

A service fails with certificate verify failed and you suspect the CA was distrusted:

# 1. Identify the CA from the failing certificate
openssl x509 -in /etc/pki/tls/certs/server.crt -noout -issuer
# issuer=CN = Internal Corp CA, O = CorpCo

# 2. Check if the issuer is trusted
trust list --filter=ca-anchors | grep -A 3 "Internal Corp CA"
# No output means it's NOT in the trusted anchors

# 3. Check if it's actively distrusted
trust list --filter=blacklist | grep -A 3 "Internal Corp CA"   # RHEL 7/8
trust list --filter=blocklist | grep -A 3 "Internal Corp CA"   # RHEL 9+
# If found here, the CA is explicitly distrusted

# 4. Check if it's present at all
trust list | grep -A 3 "Internal Corp CA"
# If not found anywhere, it was never added to the trust store

# 5. If distrusted, find the source file (checks both RHEL 7/8 and 9+ paths)
find /etc/pki/ca-trust/source/blacklist/ \
     /etc/pki/ca-trust/source/blocklist/ \
     /usr/share/pki/ca-trust-source/blacklist/ \
     /usr/share/pki/ca-trust-source/blocklist/ \
     -type f 2>/dev/null | while read f; do
    if openssl x509 -in "$f" -noout -subject 2>/dev/null | grep -qi "Internal Corp CA"; then
        echo "FOUND: $f"
    fi
done

# 6. If NOT found in any distrust directory, check for trust format conflicts:
#    look for a plain PEM in anchors/ that duplicates a cert in the system bundle
for f in /etc/pki/ca-trust/source/anchors/*; do
    [ -f "$f" ] || continue
    if openssl x509 -in "$f" -noout -subject 2>/dev/null | grep -qi "Internal Corp CA"; then
        echo "DUPLICATE IN ANCHORS: $f"
        echo "  Format: $(head -1 "$f")"
        echo "  Check if the same cert exists in ca-bundle.trust.p11-kit with different trust format"
    fi
done

# 7. Also check the main p11-kit bundle
grep -B 2 -A 10 "Internal Corp CA" \
    /usr/share/pki/ca-trust-source/ca-bundle.trust.p11-kit

Restoring Trust for a Distrusted Certificate

If you determine a certificate was incorrectly distrusted:

# If the distrust entry is in /etc/pki/ (admin-managed)
# RHEL 7/8:
sudo rm /etc/pki/ca-trust/source/blacklist/the-cert.crt
# RHEL 9+:
sudo rm /etc/pki/ca-trust/source/blocklist/the-cert.crt

sudo update-ca-trust

# If the distrust comes from the ca-certificates package, you need to
# override it by adding the certificate as a trusted anchor:
sudo cp the-cert.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust
# Note: this works because admin anchors override package-level distrust
# for certificates matched by DER content.

# Verify the certificate is now trusted
trust list --filter=ca-anchors | grep -A 3 "The Cert Label"

Warning: Overriding a Mozilla/Red Hat distrust decision should only be done if you have a specific business reason and understand the security implications. CAs are distrusted for cause (compromise, mis-issuance, policy violations).

6.8 Advanced Debugging with trust and p11-kit

Inspecting Individual Trust Objects

# Dump the full PKCS#11 attributes for a specific certificate
trust dump --filter="pkcs11:id=%DE%28%F4%A4..." 

# List all attributes of all trust objects (verbose, large output)
trust dump

Verifying the Extraction Pipeline

If you suspect update-ca-trust is not producing the expected output:

# Run the extraction manually with verbose output
p11-kit extract --format=pem-bundle \
    --filter=ca-anchors \
    --purpose=server-auth \
    /tmp/test-tls-bundle.pem

# Compare with the system bundle
diff <(sort /tmp/test-tls-bundle.pem) \
     <(sort /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem)

# Run with p11-kit debug logging
P11_KIT_DEBUG=all update-ca-trust 2>&1 | head -100

Checking Certificate Chain Completeness

# Verify a server certificate against the system trust store
openssl verify -CAfile /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem server.crt

# If the chain has intermediates, include them:
openssl verify -CAfile /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem \
    -untrusted intermediate.crt server.crt

# Show the full chain that OpenSSL would build:
openssl verify -show_chain -CAfile /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem server.crt

Cross-checking the OpenSSL Bundle

The OpenSSL “trusted certificate” format (ca-bundle.trust.crt) includes embedded trust/reject attributes that are distinct from the plain PEM bundle. You can inspect them:

# Show trust attributes for certificates in the OpenSSL bundle
openssl x509 -in /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt \
    -noout -text -trustout 2>/dev/null | grep -A 2 "Trusted Uses\|Rejected Uses"

Checking the Java Keystore

# List all entries in the Java trust store
keytool -list -cacerts -storepass changeit 2>/dev/null | grep "trustedCertEntry" | wc -l

# Search for a specific CA
keytool -list -cacerts -storepass changeit 2>/dev/null | grep -i "company"

6.9 Troubleshooting Trust Issues

Systematic Approach

Symptom: Certificate verification failed

Step 1: Identify what certificate is failing and who issued it

# Get the issuer chain from a remote server
openssl s_client -connect server.example.com:443 -showcerts </dev/null 2>/dev/null | \
    openssl x509 -noout -issuer -subject

# Or from a local certificate file
openssl x509 -in server.crt -noout -issuer -subject -serial

Step 2: Check if the issuer CA is in the trust store

trust list --filter=ca-anchors | grep -i "ISSUER_CN_HERE"

Step 3: Check if the issuer CA is distrusted

trust list --filter=blacklist | grep -i "ISSUER_CN_HERE"    # RHEL 7/8
trust list --filter=blocklist | grep -i "ISSUER_CN_HERE"    # RHEL 9+

Step 4: If missing, add it

sudo cp issuer-ca.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

Step 5: If distrusted, find the source and decide on action

# Trace the distrust source (see Section 6.7)
# Checks both RHEL 7/8 (blacklist) and RHEL 9+ (blocklist) paths
find /etc/pki/ca-trust/source/blacklist/ \
     /etc/pki/ca-trust/source/blocklist/ \
     /usr/share/pki/ca-trust-source/blacklist/ \
     /usr/share/pki/ca-trust-source/blocklist/ \
     -name '*.crt' -o -name '*.pem' 2>/dev/null | while read f; do
    openssl x509 -in "$f" -noout -subject 2>/dev/null
done

Common Trust Store Problems

Quick Reference

┌───────────────────────────────────────────────────────────────────────────────┐
│ RHEL TRUST STORE QUICK REFERENCE                                              │
├───────────────────────────────────────────────────────────────────────────────┤
│                                                                               │
│  Add CA:       sudo cp ca.crt /etc/pki/ca-trust/source/anchors/               │
│                sudo update-ca-trust                                           │
│                                                                               │
│  Distrust:     sudo cp bad.crt /etc/pki/ca-trust/source/blacklist/            │
│                (RHEL 7/8) or .../source/blocklist/ (RHEL 9+)                  │
│                sudo update-ca-trust                                           │
│                                                                               │
│  Verify:       trust list --filter=ca-anchors | grep "CA Name"                │
│                openssl verify -CAfile /etc/pki/ca-trust/extracted/            │
│                    pem/tls-ca-bundle.pem cert.crt                             │
│                                                                               │
│  Distrusted:   trust list --filter=blacklist  (RHEL 7/8)                      │
│                trust list --filter=blocklist  (RHEL 9+)                       │
│                                                                               │
│  Duplicates:   trust list --filter=ca-anchors | grep "label:" |               │
│                    sort | uniq -c | sort -rn                                  │
│                                                                               │
│  Compare:      openssl x509 -in a.crt -outform DER | sha256sum                │
│                openssl x509 -in b.crt -outform DER | sha256sum                │
│                                                                               │
│  Debug:        P11_KIT_DEBUG=all update-ca-trust 2>&1                         │
│                                                                               │
│  Source dirs:  /etc/pki/ca-trust/source/{anchors,blacklist|blocklist}/        │
│                /usr/share/pki/ca-trust-source/{anchors,blacklist|blocklist}/  │
│                                                                               │
│  Extracted:    /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem              │
│                /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt        │
│                /etc/pki/ca-trust/extracted/java/cacerts                       │
│                                                                               │
│  Pipeline:     update-ca-trust = update-ca-trust extract (identical)          │
│                                                                               │
│  Priority:     blacklist/blocklist > anchors                                  │
│                /etc/pki/ > /usr/share/pki/                                    │
│                .p11-kit attributes > plain PEM defaults                       │
│                                                                               │
│  DANGER:       Never add a plain "BEGIN CERTIFICATE" PEM to anchors/          │
│                if the same cert already exists in the system bundle           │
│                as "BEGIN TRUSTED CERTIFICATE" or .p11-kit format.             │
│                Empty trust (no uses) = "trusted for nothing" = DISTRUSTED.    │
│                Rejected uses + plain PEM → also DISTRUSTED.                   │
└───────────────────────────────────────────────────────────────────────────────┘

🧪 Hands-On Lab

Lab 05: Trust Store Management

Add custom CAs to the system trust store, manage trust attributes, detect duplicates, and debug distrusted certificates.

• 📁 Location: labs/en_US/05-trust-store/
• ⏱️ Time: 45 minutes
• 🎯 Level: Intermediate

Chapter Navigation

Chapter 7: Digital Signatures & Verification on RHEL

How Trust Works: Learn how digital signatures enable certificate validation on RHEL systems.

7.1 Cryptographic Hash Functions

Properties:

1. Deterministic
2. Pre-image resistance
3. Collision resistance
4. Avalanche effect

Popular algorithms: SHA-256, SHA-3, BLAKE2.

7.2 Building Signatures

1. Calculate hash of message.
2. Encrypt hash with private key → signature.
3. Receiver decrypts signature with public key and compares with their own hash.

7.3 Certificate Fingerprints

A fingerprint is simply the hash of the DER-encoded certificate, used to uniquely identify it, e.g.,

openssl x509 -in server.crt -noout -fingerprint -sha256

7.4 Lab: Sign & Verify a File

# sign
openssl dgst -sha256 -sign rsa.key.pem -out report.sig report.pdf
# verify
openssl dgst -sha256 -verify rsa.pub.pem -signature report.sig report.pdf

7.5 Summary

Signatures tie data to identities; hashes ensure integrity. Together they underpin certificate validation and all PKI operations.

7.6 Signature Algorithms on RHEL

Approved by RHEL Version

Critical: RHEL 9+ blocks SHA-1 and MD5 for security!

Verifying Certificates on RHEL

# Verify certificate chain
openssl verify /etc/pki/tls/certs/server.crt

# Verify against specific CA
openssl verify -CAfile /etc/pki/tls/certs/ca-bundle.crt server.crt

# Check signature algorithm
openssl x509 -in server.crt -noout -text | grep "Signature Algorithm"
# Must be SHA-256+ on RHEL 8+

Quick Reference

┌──────────────────────────────────────────────────────────┐
│ DIGITAL SIGNATURES ON RHEL                               │
├──────────────────────────────────────────────────────────┤
│ Purpose:      Prove authenticity and integrity           │
│ How:          Hash + Private key = Signature             │
│ Verify:       Signature + Public key = Original hash     │
│                                                          │
│ Approved:     SHA-256, SHA-384, SHA-512                  │
│ Deprecated:   SHA-1 (blocked on RHEL 9+)                 │
│ Blocked:      MD5 (blocked on RHEL 9+)                   │
│                                                          │
│ Verify cert:  openssl verify cert.crt                    │
│ Check alg:    openssl x509 -noout -text | grep Signature │
│ Fingerprint:  openssl x509 -noout -fingerprint -sha256   │
└──────────────────────────────────────────────────────────┘

🧪 Hands-On Lab

Lab 03: Digital Signatures

Sign files, verify signatures, and detect tampering

• 📁 Location: labs/en_US/03-digital-signatures/
• ⏱️ Time: 20 minutes
• 🎯 Level: Beginner

Chapter Navigation

Chapter 8: RHEL Versions & Certificate Evolution

Learning Objective: Understand how certificate management differs across RHEL 7, 8, 9, and 10 so you can quickly identify version-specific behaviors when troubleshooting.

8.1 Why RHEL Version Matters

When troubleshooting certificate issues on RHEL, the first question should always be: “What RHEL version am I on?”

Certificate management has evolved significantly across RHEL versions. What works on RHEL 7 may fail on RHEL 9. Understanding these differences is crucial for:

• ✅ Choosing the right troubleshooting approach
• ✅ Identifying version-specific errors
• ✅ Planning migrations
• ✅ Writing compatible automation scripts

8.2 Quick Version Check

# Method 1: Check /etc/redhat-release
cat /etc/redhat-release
# Output examples:
#   Red Hat Enterprise Linux Server release 7.9 (Maipo)
#   Red Hat Enterprise Linux release 8.10 (Ootpa)
#   Red Hat Enterprise Linux release 9.8 (Plow)
#   Red Hat Enterprise Linux release 10.2 (Coughlan)

# Method 2: Use rpm
rpm -q --queryformat '%{VERSION}\n' redhat-release

# Method 3: Check OpenSSL version (indirect but useful)
openssl version
# RHEL 7: OpenSSL 1.0.2k family
# RHEL 8: OpenSSL 1.1.1 family
# RHEL 9: OpenSSL 3.x
# RHEL 10: OpenSSL 3.x

8.3 RHEL Version Overview

Source: Red Hat Product Life Cycle

Use the product life cycle page for exact EUS/ELS/add-on details; the table above is a major-version planning view, not a contract summary.

8.4 Major Differences Summary

RHEL 7 (Legacy)

Characteristics:

• ✅ Stable, well-understood
• ✅ Maximum compatibility
• ⚠️ TLS 1.0/1.1 enabled by default
• ⚠️ Weak ciphers allowed
• ⚠️ Manual certificate management

Package Family: openssl-1.0.2k*

When You’ll See It:

• Legacy systems not yet migrated
• Applications requiring old TLS versions
• Conservative environments

Key Command:

# Generate key (RHEL 7 style)
openssl genrsa -out server.key 2048

RHEL 8 (Widely Deployed)

Characteristics:

• ✅ System-wide crypto-policies (game changer!)
• ✅ TLS 1.2+ by default
• ✅ certmonger for auto-renewal
• ✅ Modern cipher suites
• ⚠️ Breaking changes from RHEL 7

Package Family: openssl-1.1.1*

Key Innovation - Crypto-Policies:

# View current policy
update-crypto-policies --show
# DEFAULT, LEGACY, FUTURE, or FIPS

# System-wide security control!
sudo update-crypto-policies --set FUTURE

When You’ll See It:

• Most enterprise deployments
• Modern applications
• FreeIPA environments

Key Command:

# Generate key (RHEL 8 modern style)
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out server.key

RHEL 9 (Modern Standard)

Characteristics:

• ✅ OpenSSL 3.5.5 with provider architecture
• ✅ TLS 1.2+ mandatory
• ✅ Enhanced crypto-policies
• ✅ Stricter certificate validation
• ⚠️ OpenSSL 3.x API changes
• ⚠️ Legacy algorithms disabled

Package Family: openssl-3*

Major Change - Provider Architecture:

# List crypto providers
openssl list -providers
# default, fips, legacy, base

# Legacy algorithms require explicit provider
openssl md5 -provider legacy file.txt

When You’ll See It:

• New deployments
• Security-conscious environments
• Latest application versions

Key Command:

# Generate EC key (RHEL 9)
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out ec.key

RHEL 10 (Current Release)

Characteristics:

• ✅ Same OpenSSL 3.5.5 as RHEL 9.8
• ✅ Continued security hardening
• ✅ Post-quantum cryptography preparation
• ✅ Enhanced container certificate support
• ⚠️ Even stricter defaults
• ⚠️ Additional legacy removals

Package Family: openssl-3*

Note: RHEL 10.0 GA was May 20, 2025. Features and capabilities may evolve across minor versions (10.1, 10.2, etc.). Always consult official documentation for your specific RHEL 10.x release.

Key Focus:

• Quantum-resistant cryptography foundation
• Modern security practices
• Container and cloud-native workloads

When You’ll See It:

• Brand new deployments
• Cutting-edge security requirements
• Future-proofing initiatives

8.5 Critical Version Differences

TLS Version Support

Certificate Tools Availability

Key Cipher/Algorithm Changes

8.6 Common Version-Specific Issues

RHEL 7 Issues

# Problem: Old cipher suites accepted
# Impact: Security vulnerabilities
# Solution: Manual Apache/NGINX cipher configuration

RHEL 8 Issues

# Problem: Application fails after migration from RHEL 7
# Reason: TLS 1.0/1.1 disabled by default
# Quick Fix: Temporarily use LEGACY policy (not recommended long-term)
sudo update-crypto-policies --set LEGACY

# Better Fix: Update application to support TLS 1.2+

RHEL 9 Issues

# Problem: OpenSSL commands fail with provider errors
# Reason: OpenSSL 3.x provider architecture
# Fix: Specify provider explicitly
openssl md5 -provider legacy file.txt

# Problem: SHA-1 certificates rejected
# Reason: Stricter validation
# Fix: Reissue certificates with SHA-256+

RHEL 10 Issues

# Problem: Even stricter defaults than RHEL 9
# Impact: Legacy certificates may fail validation
# Solution: Ensure all certificates use modern algorithms
#           Check RHEL 10.x specific documentation for your minor version

8.7 Migration Impact

RHEL 7 → RHEL 8

Certificate Impact: MODERATE

• TLS 1.0/1.1 disabled
• Weak ciphers removed
• certmonger integration required for automation

Action Required:

1. Audit TLS versions in use
2. Update cipher configurations
3. Test applications with TLS 1.2+
4. Consider crypto-policies

RHEL 8 → RHEL 9

Certificate Impact: HIGH

• OpenSSL 3.x API changes
• Legacy algorithm removal
• Stricter certificate validation
• Provider architecture changes

Action Required:

1. Test all certificate operations
2. Update custom scripts using OpenSSL
3. Validate certificate chain integrity
4. Check for SHA-1 usage

RHEL 9 → RHEL 10

Certificate Impact: LOW-MODERATE

• Same OpenSSL base (3.5.5)
• Incremental hardening
• Policy refinements

Action Required:

1. Review RHEL 10.x documentation
2. Test crypto-policy compatibility
3. Validate modern algorithm usage

8.8 Choosing the Right Approach

For Troubleshooting

# Always start with version check
cat /etc/redhat-release

# Then check OpenSSL version
openssl version

# For RHEL 8+: Check crypto-policy
update-crypto-policies --show 2>/dev/null || echo "Pre-RHEL 8"

Quick Reference Decision Tree

Is it RHEL 7?
├─ YES → Check for legacy TLS/cipher issues
│        Manual configuration likely needed
│        Consider migration planning
│
└─ NO → Is it RHEL 8?
    ├─ YES → Check crypto-policies first!
    │        Use certmonger for automation
    │        Consider RHEL 9 upgrade
    │
    └─ NO → Is it RHEL 9 or 10?
        └─ YES → Check OpenSSL 3.x provider issues
                 Verify modern algorithms in use
                 Leverage enhanced tooling

8.9 Key Takeaways

1. Always check RHEL version first when troubleshooting
2. RHEL 8 introduced crypto-policies - game changer for certificate management
3. RHEL 9 uses OpenSSL 3.x - significant API and behavior changes
4. RHEL 10 continues RHEL 9 foundation - incremental improvements
5. Legacy algorithms progressively removed across versions
6. Migration testing is critical - certificate behavior changes significantly

8.10 What’s Next?

Now that you understand RHEL version differences, we’ll dive deeper into:

• Chapter 9: RHEL 7 Certificate Management (detailed)
• Chapter 10: RHEL 8 & Crypto-Policies (detailed)
• Chapter 11: RHEL 9 Modern Security (detailed)
• Chapter 12: RHEL 10 Current Features (detailed)

Quick Reference Card

┌────────────────────────────────────────────────────────────┐
│ RHEL VERSION QUICK REFERENCE                               │
├─────────────┬───────────┬──────────────┬───────────────────┤
│ RHEL 7      │ 1.0.2k    │ Manual       │ Legacy-friendly   │
│ RHEL 8      │ 1.1.1k    │ Crypto-pols  │ Widely deployed   │
│ RHEL 9      │ 3.5.5     │ OpenSSL 3.x  │ Modern secure     │
│ RHEL 10     │ 3.5.5     │ Hardened     │ Future-ready      │
└─────────────┴───────────┴──────────────┴───────────────────┘

Check version:  cat /etc/redhat-release
Check OpenSSL:  openssl version
Check policy:   update-crypto-policies --show  (RHEL 8+)

Chapter Navigation

Chapter 9: RHEL 7 Certificate Management

Legacy but Important: RHEL 7 reached end of maintenance in June 2024, but many enterprises still run it. Learn how certificate management works on RHEL 7.

9.1 RHEL 7 Overview

Release: June 10, 2014 Maintenance Support Ended: June 30, 2024 Extended Life Cycle Support: Available through 2028

Key Characteristics:

• OpenSSL Version: 1.0.2k-26 (package: openssl-1.0.2k-26.el7_9.x86_64)
• Default TLS: TLS 1.0, 1.1, 1.2 all enabled
• Trust Store: /etc/pki/ca-trust/extracted/
• Management Approach: Primarily manual
• Crypto-Policies: Not available (RHEL 8+ feature)

Note: If you’re still on RHEL 7, plan migration to RHEL 8 or 9. Security updates are limited.

9.2 OpenSSL 1.0.2k Specifics

Version Check

# Check OpenSSL version on RHEL 7
openssl version
# OpenSSL 1.0.2k-fips  12 Jan 2017

# Check package
rpm -q openssl
# openssl-1.0.2k-26.el7_9.x86_64

Key Features and Limitations

Features:

• ✅ TLS 1.0, 1.1, 1.2 support
• ✅ Stable and well-tested
• ✅ Wide compatibility
• ✅ RSA, ECC, DSA key types

Limitations:

• ❌ No TLS 1.3 support
• ❌ Older command syntax (genrsa vs genpkey)
• ❌ Weaker default ciphers
• ❌ Limited modern cipher suites

Command Syntax (RHEL 7 Style)

#============================================#
# GENERATE RSA KEY (RHEL 7)
#============================================#

# Old style (common on RHEL 7)
openssl genrsa -out server.key 2048

# With passphrase protection
openssl genrsa -aes256 -out server.key 2048

# Remove passphrase from key
openssl rsa -in server.key -out server-nopass.key


#============================================#
# GENERATE CSR (RHEL 7)
#============================================#

# Basic CSR
openssl req -new -key server.key -out server.csr

# With subject specified
openssl req -new -key server.key -out server.csr \
  -subj "/C=US/ST=State/L=City/O=Company/CN=server.example.com"

# ⚠️ Note: SANs are harder to add with RHEL 7 OpenSSL
# Need config file for SANs


#============================================#
# VIEW CERTIFICATE
#============================================#

# Full details
openssl x509 -in server.crt -noout -text

# Just expiration
openssl x509 -in server.crt -noout -dates

# Just subject
openssl x509 -in server.crt -noout -subject

9.3 Trust Store Management on RHEL 7

Adding Custom CAs

#============================================#
# ADD CUSTOM CA (RHEL 7)
#============================================#

# Step 1: Copy CA certificate to anchors directory
sudo cp corporate-ca.crt /etc/pki/ca-trust/source/anchors/

# Step 2: Update trust store
sudo update-ca-trust extract

# Step 3: Verify
trust list | grep -i "corporate"

# Verify applications use it
openssl verify -CAfile /etc/pki/tls/certs/ca-bundle.crt test-cert.crt

Trust Store Locations (RHEL 7)

/etc/pki/ca-trust/
├── source/
│   └── anchors/                   ← Add custom CAs here
│
└── extracted/
    ├── pem/
    │   └── tls-ca-bundle.pem      ← OpenSSL, Python, Ruby
    ├── openssl/
    │   └── ca-bundle.trust.crt    ← OpenSSL specific
    └── java/
        └── cacerts                ← Java applications

9.4 Service Configuration (RHEL 7 Approach)

Apache HTTPS on RHEL 7

#============================================#
# APACHE SSL/TLS SETUP (RHEL 7)
#============================================#

# Install Apache with SSL
sudo yum install httpd mod_ssl -y

# Generate certificate and key
sudo openssl genrsa -out /etc/pki/tls/private/server.key 2048
sudo openssl req -new -key /etc/pki/tls/private/server.key \
  -out /tmp/server.csr \
  -subj "/CN=$(hostname -f)"

# Get certificate from CA (or self-signed for testing)
sudo openssl x509 -req -days 365 -in /tmp/server.csr \
  -signkey /etc/pki/tls/private/server.key \
  -out /etc/pki/tls/certs/server.crt

# Configure Apache (/etc/httpd/conf.d/ssl.conf)
sudo vi /etc/httpd/conf.d/ssl.conf
# Set:
#   SSLCertificateFile /etc/pki/tls/certs/server.crt
#   SSLCertificateKeyFile /etc/pki/tls/private/server.key
#
#   # Recommended: Disable weak TLS versions
#   SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
#
#   # Recommended: Strong ciphers only
#   SSLCipherSuite HIGH:!aNULL:!MD5:!3DES:!RC4

# Start Apache
sudo systemctl enable httpd
sudo systemctl start httpd

# Test
curl -vk https://localhost/

NGINX on RHEL 7

#============================================#
# NGINX SSL/TLS SETUP (RHEL 7)
#============================================#

# Install NGINX (from EPEL)
sudo yum install epel-release -y
sudo yum install nginx -y

# Generate certificate
sudo openssl genrsa -out /etc/pki/tls/private/nginx.key 2048
sudo openssl req -new -x509 -days 365 \
  -key /etc/pki/tls/private/nginx.key \
  -out /etc/pki/tls/certs/nginx.crt \
  -subj "/CN=$(hostname -f)"

# Configure NGINX (/etc/nginx/nginx.conf)
# Add to server block:
#   listen 443 ssl;
#   ssl_certificate /etc/pki/tls/certs/nginx.crt;
#   ssl_certificate_key /etc/pki/tls/private/nginx.key;
#
#   # Recommended
#   ssl_protocols TLSv1.2;
#   ssl_ciphers HIGH:!aNULL:!MD5;

# Start NGINX
sudo systemctl enable nginx
sudo systemctl start nginx

9.5 Manual Certificate Renewal (RHEL 7)

No crypto-policies, no automatic tools - everything is manual!

Renewal Process

#============================================#
# MANUAL RENEWAL PROCESS (RHEL 7)
#============================================#

# Step 1: Check expiration (set calendar reminder)
openssl x509 -in /etc/pki/tls/certs/server.crt -noout -dates

# Step 2: Generate new CSR (reuse existing key)
openssl req -new -key /etc/pki/tls/private/server.key \
  -out /tmp/server-renewal.csr \
  -subj "/CN=server.example.com"

# Step 3: Submit CSR to CA

# Step 4: Receive new certificate from CA

# Step 5: Backup old certificate
sudo cp /etc/pki/tls/certs/server.crt \
     /etc/pki/tls/certs/server.crt.$(date +%Y%m%d).old

# Step 6: Install new certificate
sudo cp new-server.crt /etc/pki/tls/certs/server.crt
sudo chmod 644 /etc/pki/tls/certs/server.crt

# Step 7: Reload service
sudo systemctl reload httpd

# Step 8: Test
curl -v https://localhost/
openssl s_client -connect localhost:443

Tracking Certificate Renewals

#============================================#
# CREATE RENEWAL TRACKING (RHEL 7)
#============================================#

# Cron job to check expiration
cat > /etc/cron.weekly/check-cert-expiration << 'EOF'
#!/bin/bash
# Check certificates expiring in 60 days

for cert in /etc/pki/tls/certs/*.crt; do
  [ -f "$cert" ] || continue

  if ! openssl x509 -in "$cert" -noout -checkend $((86400*60)); then
    echo "⚠️ $cert expires within 60 days!"
    echo "$cert" | mail -s "Certificate Expiring Soon" admin@example.com
  fi
done
EOF

chmod +x /etc/cron.weekly/check-cert-expiration

9.6 Common RHEL 7 Certificate Issues

Issue 1: TLS 1.0/1.1 Deprecated

Problem: Modern clients reject TLS 1.0/1.1

Symptoms:

curl: (35) error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure

Fix:

# Update Apache to disable old TLS versions
# /etc/httpd/conf.d/ssl.conf
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1

# Restart Apache
sudo systemctl restart httpd

Issue 2: Weak Ciphers

Problem: PCI/Security scans flag weak ciphers

Fix:

# Apache: Use strong ciphers only
# /etc/httpd/conf.d/ssl.conf
SSLCipherSuite HIGH:!aNULL:!MD5:!3DES:!RC4:!EXPORT
SSLHonorCipherOrder on

# Test
openssl s_client -connect localhost:443 -cipher '3DES'
# Should fail if 3DES is disabled

Issue 3: Missing SANs

Problem: Modern browsers require Subject Alternative Names

RHEL 7 Challenge: SANs are harder to add with OpenSSL 1.0.2

Solution: Use config file

# Create OpenSSL config
cat > /tmp/san.cnf << EOF
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req

[req_distinguished_name]
CN = server.example.com

[v3_req]
subjectAltName = @alt_names

[alt_names]
DNS.1 = server.example.com
DNS.2 = www.example.com
IP.1 = 10.0.0.100
EOF

# Generate CSR with SANs
openssl req -new -key server.key -out server.csr -config /tmp/san.cnf

# Verify SANs in CSR
openssl req -in server.csr -noout -text | grep -A3 "Subject Alternative Name"

9.7 certmonger on RHEL 7

Available: Yes (basic version)

#============================================#
# CERTMONGER ON RHEL 7
#============================================#

# Install
sudo yum install certmonger -y
sudo systemctl enable certmonger
sudo systemctl start certmonger

# Request certificate from FreeIPA
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/web.crt \
  -k /etc/pki/tls/private/web.key \
  -K host/$(hostname -f)@REALM

# List tracked certificates
sudo getcert list

# Check specific certificate status
sudo getcert list -f /etc/pki/tls/certs/web.crt

# Monitor certmonger logs
sudo tail -f /var/log/messages | grep certmonger

RHEL 7 Limitations:

• No ACME support (Let’s Encrypt requires manual certbot)
• Less detailed status output
• Fewer post-save command options

9.8 Migration Considerations

When to Migrate from RHEL 7

You should migrate if:

• ✅ Support ended (June 2024) and you need updates
• ✅ Need TLS 1.3 support
• ✅ Want crypto-policies for easier management
• ✅ Require modern security features
• ✅ Compliance requires supported OS

Pre-Migration Certificate Tasks

#============================================#
# RHEL 7 CERTIFICATE PRE-MIGRATION AUDIT
#============================================#

# 1. List all certificates
find /etc/pki/tls/ -name "*.crt" -o -name "*.key"

# 2. Check expirations
for cert in /etc/pki/tls/certs/*.crt; do
  echo "=== $cert ==="
  openssl x509 -in "$cert" -noout -subject -dates
  echo ""
done

# 3. Check signature algorithms (SHA-1 won't work on RHEL 8+)
for cert in /etc/pki/tls/certs/*.crt; do
  SIG=$(openssl x509 -in "$cert" -noout -text | grep "Signature Algorithm" | head -2)
  echo "$cert: $SIG"
done | grep -i sha1
# If any found, reissue before migration!

# 4. Document custom CAs
ls -l /etc/pki/ca-trust/source/anchors/

# 5. Export certificates and keys
tar czf rhel7-certificates-backup-$(date +%Y%m%d).tar.gz \
  /etc/pki/tls/certs/*.crt \
  /etc/pki/tls/private/*.key \
  /etc/pki/ca-trust/source/anchors/*

9.9 Common RHEL 7 Workflows

Workflow 1: Manual Apache HTTPS Setup

# Complete workflow from scratch

# 1. Install Apache with SSL
sudo yum install httpd mod_ssl -y

# 2. Generate private key
sudo openssl genrsa -out /etc/pki/tls/private/$(hostname -s).key 2048

# 3. Set key permissions
sudo chmod 600 /etc/pki/tls/private/$(hostname -s).key

# 4. Create CSR
sudo openssl req -new \
  -key /etc/pki/tls/private/$(hostname -s).key \
  -out /tmp/$(hostname -s).csr \
  -subj "/C=US/O=Company/CN=$(hostname -f)"

# 5. Submit CSR to CA, wait for certificate

# 6. Install certificate
sudo cp $(hostname -s).crt /etc/pki/tls/certs/

# 7. Configure Apache
sudo vi /etc/httpd/conf.d/ssl.conf
# Edit:
#   SSLCertificateFile /etc/pki/tls/certs/$(hostname -s).crt
#   SSLCertificateKeyFile /etc/pki/tls/private/$(hostname -s).key
#   SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
#   SSLCipherSuite HIGH:!aNULL:!MD5:!3DES

# 8. Test configuration
sudo apachectl configtest

# 9. Open firewall
sudo firewall-cmd --add-service=https --permanent
sudo firewall-cmd --reload

# 10. Start Apache
sudo systemctl enable httpd
sudo systemctl start httpd

# 11. Test
curl -vk https://$(hostname -f)/

Workflow 2: FreeIPA Integration

#============================================#
# FREEIPA CERTIFICATE WORKFLOW (RHEL 7)
#============================================#

# Prerequisites: System must be IPA-enrolled
ipa-client-install

# Install certmonger
sudo yum install certmonger -y
sudo systemctl enable certmonger
sudo systemctl start certmonger

# Request certificate for Apache
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/$(hostname -s).crt \
  -k /etc/pki/tls/private/$(hostname -s).key \
  -K host/$(hostname -f)@REALM.EXAMPLE.COM \
  -D $(hostname -f)

# Check status
sudo getcert list

# Wait for MONITORING status (certificate issued)

# Configure Apache to use cert
# /etc/httpd/conf.d/ssl.conf

# Reload Apache when cert renews
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/$(hostname -s).crt \
  -k /etc/pki/tls/private/$(hostname -s).key \
  -K host/$(hostname -f)@REALM \
  -C "systemctl reload httpd"

9.10 Troubleshooting RHEL 7 Certificates

Diagnostic Commands

#============================================#
# RHEL 7 CERTIFICATE DIAGNOSTICS
#============================================#

# Check OpenSSL version
openssl version

# Test HTTPS locally
openssl s_client -connect localhost:443

# Check Apache SSL configuration
sudo apachectl -t -D DUMP_VHOSTS | grep 443

# View Apache SSL errors
sudo tail -f /var/log/httpd/ssl_error_log

# Check SELinux denials
sudo grep AVC /var/log/audit/audit.log | grep cert

# Check file permissions
ls -lZ /etc/pki/tls/certs/*.crt
ls -lZ /etc/pki/tls/private/*.key

# Verify certificate/key pair
openssl x509 -noout -modulus -in /etc/pki/tls/certs/server.crt | openssl md5
openssl rsa -noout -modulus -in /etc/pki/tls/private/server.key | openssl md5
# MD5 hashes should match

Common RHEL 7 Errors

9.11 Security Hardening on RHEL 7

Recommended Configuration

#============================================#
# APACHE SSL/TLS HARDENING (RHEL 7)
#============================================#

# /etc/httpd/conf.d/ssl.conf

# Disable old protocols
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1

# Strong ciphers only
SSLCipherSuite ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:HIGH:!aNULL:!MD5:!RC4:!3DES:!DES

# Honor server cipher preference
SSLHonorCipherOrder on

# Enable HSTS (HTTP Strict Transport Security)
Header always set Strict-Transport-Security "max-age=31536000"

# OCSP Stapling (not available in RHEL 7 OpenSSL 1.0.2 by default)
# Available in some backports

# Perfect Forward Secrecy
SSLCipherSuite ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256

9.12 Migration Path to RHEL 8+

Certificate-Specific Migration Steps

#============================================#
# PREPARE CERTIFICATES FOR MIGRATION
#============================================#

# 1. Verify all certificates use SHA-256+ (no SHA-1 or MD5)
for cert in /etc/pki/tls/certs/*.crt; do
  openssl x509 -in "$cert" -noout -text | grep "Signature Algorithm"
done | grep -i sha1 && echo "⚠️ SHA-1 certificates found! Reissue before migration!"

# 2. Verify key sizes (2048+ bits)
for cert in /etc/pki/tls/certs/*.crt; do
  SIZE=$(openssl x509 -in "$cert" -noout -text | grep "Public-Key" | grep -oP '\d+')
  if [ "$SIZE" -lt 2048 ]; then
    echo "⚠️ $cert: Key too small ($SIZE bits)"
  fi
done

# 3. Backup everything
tar czf rhel7-certs-$(hostname)-$(date +%Y%m%d).tar.gz \
  /etc/pki/tls/ \
  /etc/pki/ca-trust/source/anchors/ \
  /etc/httpd/conf.d/ssl.conf \
  /etc/nginx/nginx.conf

# 4. Document certificate inventory
./generate-cert-inventory.sh > cert-inventory-pre-migration.csv

# 5. Test TLS 1.2 compatibility
# Ensure all services work with TLS 1.2 only

9.13 When RHEL 7 Makes Sense

Still Using RHEL 7? Consider:

Reasons to Stay (Temporarily):

• Extended Life Cycle Support contract active
• Critical legacy applications requiring TLS 1.0/1.1
• Migration planned for near future
• Testing RHEL 8/9 in parallel

Reasons to Migrate:

• ✅ Extended maintenance ended June 2024
• ✅ No crypto-policies (harder to manage)
• ✅ No TLS 1.3
• ✅ Security updates limited
• ✅ Modern applications dropping TLS 1.0/1.1 support

9.14 Key Takeaways

1. RHEL 7 is manual - No crypto-policies, careful configuration needed
2. OpenSSL 1.0.2k - Older syntax, no TLS 1.3
3. TLS 1.0/1.1 enabled by default - Disable them manually
4. SHA-1 still works - But won’t after migration to RHEL 8+
5. certmonger available - But basic compared to RHEL 8+
6. Plan migration - RHEL 7 support is ending
7. Document everything - Makes migration easier

Quick Reference

┌───────────────────────────────────────────────────────┐
│ RHEL 7 CERTIFICATE QUICK REFERENCE                    │
├───────────────────────────────────────────────────────┤
│ OpenSSL:   1.0.2k-26                                  │
│ TLS:       1.0, 1.1, 1.2 (no 1.3)                     │
│ Policy:    Manual configuration (no crypto-policies)  │
│                                                       │
│ Generate:  openssl genrsa -out key.pem 2048           │
│ CSR:       openssl req -new -key key.pem -out req.csr │
│ View:      openssl x509 -in cert.crt -noout -text     │
│ Test:      openssl s_client -connect host:443         │
│                                                       │
│ Harden:    SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1     │
│            SSLCipherSuite HIGH:!aNULL:!MD5:!3DES      │
└───────────────────────────────────────────────────────┘

Chapter Navigation

Chapter 10: RHEL 8 & Crypto-Policies

Game Changer: RHEL 8 introduced crypto-policies, revolutionizing how certificates and cryptography are managed system-wide. This is the most important feature to understand for RHEL 8.

10.1 What Changed in RHEL 8?

Release: May 7, 2019 Support Until: May 31, 2029 Current Version: RHEL 8.10 (as of 2024)

Major Certificate-Related Changes:

Package: openssl-1.1.1k-14.el8_6.x86_64

10.2 Understanding Crypto-Policies

The Revolutionary Idea

RHEL 7 Problem:

❌ Configure Apache:     SSLProtocol, SSLCipherSuite
❌ Configure NGINX:      ssl_protocols, ssl_ciphers
❌ Configure Postfix:    smtpd_tls_protocols
❌ Configure OpenLDAP:   olcTLSProtocolMin
❌ Configure every application differently!

RHEL 8 Solution:

✅ Set ONE system-wide policy
✅ All applications automatically comply!

How It Works

┌────────────────────────────────────────┐
│  update-crypto-policies --set DEFAULT  │  ← Single command
└──────────────────┬─────────────────────┘
                   │
       ┌───────────┴────────────┐
       │ Crypto-Policies System │
       └───────────┬────────────┘
                   │
    ┌──────────────┼──────────────┐
    ▼              ▼              ▼
 OpenSSL        GnuTLS         NSS
 Postfix        Apache         NGINX
 OpenSSH        Kerberos       BIND
 (all apps!)    (automatic!)   (consistent!)

10.3 Available Crypto-Policies

The Four Main Policies

# Check current policy
update-crypto-policies --show

# Policies available in RHEL 8:

Policy Details

DEFAULT Policy:

TLS Versions: 1.2, 1.3
Minimum RSA/DH: 2048 bits
Minimum ECC: secp256r1 (P-256)
Ciphers: AES-GCM, ChaCha20-Poly1305, AES-CBC
Signatures: SHA-256, SHA-384, SHA-512
Blocked: MD5, SHA-1 signatures, 3DES, RC4, DSS

LEGACY Policy:

TLS Versions: 1.0, 1.1, 1.2, 1.3
Minimum RSA/DH: 1024 bits
Ciphers: Includes 3DES, weak ciphers
Signatures: Allows SHA-1
Use: Only for old systems compatibility (temporary!)

FUTURE Policy:

TLS Versions: 1.2, 1.3 (stricter ciphers)
Minimum RSA/DH: 3072 bits
Minimum ECC: secp384r1 (P-384)
Signatures: SHA-384, SHA-512 preferred
Blocked: Everything in DEFAULT, plus more

FIPS Policy:

TLS Versions: 1.2, 1.3
Algorithms: Only FIPS 140-2 approved
Requires: FIPS mode enabled
Strictest: Federal compliance requirements

10.4 Changing Crypto-Policies

Basic Policy Changes

#============================================#
# VIEW CURRENT POLICY
#============================================#

update-crypto-policies --show
# DEFAULT


#============================================#
# SET POLICY
#============================================#

# Set to FUTURE (stricter)
sudo update-crypto-policies --set FUTURE

# Set to LEGACY (less secure, for compatibility)
sudo update-crypto-policies --set LEGACY

# Set to FIPS (requires FIPS mode enabled)
sudo fips-mode-setup --enable
sudo reboot
sudo update-crypto-policies --set FIPS

# Return to DEFAULT
sudo update-crypto-policies --set DEFAULT


#============================================#
# APPLY POLICY (restart services)
#============================================#

# Crypto-policies update config files, but services must restart
sudo systemctl restart httpd nginx postfix

# Or reboot (ensures everything picks up new policy)
sudo reboot

What Happens When You Change Policy

# Example: Switching to FUTURE policy

# Before:
update-crypto-policies --show
# DEFAULT

# After:
sudo update-crypto-policies --set FUTURE

# Changes happen in:
ls -l /etc/crypto-policies/back-ends/
# opensslcnf.config      ← OpenSSL config updated
# gnutls.config          ← GnuTLS config updated
# nss.config             ← NSS config updated
# bind.config            ← BIND config updated
# ... and more

# View OpenSSL policy applied:
cat /etc/crypto-policies/back-ends/opensslcnf.config

10.5 Policy Impact on Certificates

DEFAULT Policy Impact

#============================================#
# WHAT DEFAULT POLICY ALLOWS/BLOCKS
#============================================#

# ✅ ALLOWED:
- TLS 1.2, 1.3
- RSA 2048+ bits
- AES-128-GCM, AES-256-GCM
- ChaCha20-Poly1305
- SHA-256, SHA-384, SHA-512 signatures

# ❌ BLOCKED:
- TLS 1.0, 1.1
- RSA < 2048 bits
- 3DES, RC4, DES
- MD5, SHA-1 signatures
- DSA keys
- Export ciphers

Testing Against Current Policy

#============================================#
# TEST IF YOUR CERTIFICATE WORKS
#============================================#

# Test TLS 1.2
openssl s_client -connect server.example.com:443 -tls1_2

# Test TLS 1.3
openssl s_client -connect server.example.com:443 -tls1_3

# Test specific cipher
openssl s_client -connect server.example.com:443 \
  -cipher 'ECDHE-RSA-AES256-GCM-SHA384'

# See what ciphers are available under current policy
openssl ciphers -v | head -20

10.6 OpenSSL 1.1.1 Features (RHEL 8)

New Features

#============================================#
# TLS 1.3 SUPPORT (New in RHEL 8!)
#============================================#

# Test TLS 1.3
openssl s_client -connect server.example.com:443 -tls1_3

# TLS 1.3 benefits:
# - Faster handshake
# - Forward secrecy mandatory
# - Removed outdated features


#============================================#
# MODERN KEY GENERATION
#============================================#

# Old style (still works)
openssl genrsa -out server.key 2048

# New style (preferred on RHEL 8)
openssl genpkey -algorithm RSA -out server.key \
  -pkeyopt rsa_keygen_bits:2048

# EC keys (elliptic curve)
openssl genpkey -algorithm EC -out ec.key \
  -pkeyopt ec_paramgen_curve:P-256


#============================================#
# IMPROVED CSR GENERATION
#============================================#

# CSR with SANs (much easier than RHEL 7!)
openssl req -new -key server.key -out server.csr \
  -subj "/CN=server.example.com" \
  -addext "subjectAltName=DNS:server.example.com,DNS:www.example.com,IP:10.0.0.100"

# Verify SANs
openssl req -in server.csr -noout -text | grep -A2 "Subject Alternative Name"

10.7 certmonger Enhancements in RHEL 8

Improved Features

#============================================#
# CERTMONGER ON RHEL 8
#============================================#

# Better IPA integration
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/web.crt \
  -k /etc/pki/tls/private/web.key \
  -D web.example.com \
  -K host/web.example.com@REALM \
  -C "systemctl reload httpd"  # Post-save command (improved!)

# Enhanced status output
sudo getcert list -v

# Better error reporting
sudo getcert list -f /etc/pki/tls/certs/web.crt
# Shows detailed error messages if renewal fails

RHEL 8 certmonger improvements:

• ✅ Better error messages
• ✅ Post-save command support
• ✅ Improved IPA integration
• ✅ More reliable renewal

10.8 Common RHEL 8 Scenarios

Scenario 1: Migrated from RHEL 7, TLS 1.0 App Breaks

Problem:

# Application worked on RHEL 7
# After migration to RHEL 8: connection failures

Diagnosis:

# Check crypto-policy
update-crypto-policies --show
# DEFAULT  ← TLS 1.0/1.1 disabled!

# Check application logs
journalctl -xe | grep -i tls
# "wrong version number" or "no shared cipher"

Quick Fix (Temporary):

# Use LEGACY policy to allow TLS 1.0/1.1
sudo update-crypto-policies --set LEGACY
sudo systemctl restart <service>

Proper Fix:

# Update application to support TLS 1.2+
# Or configure application specifically (opt-out of policy)

Scenario 2: Need to Support Old Clients

Problem: Windows Server 2008, Java 7 clients can’t connect

Solution:

# Option 1: LEGACY policy (not recommended long-term)
sudo update-crypto-policies --set LEGACY

# Option 2: Custom policy module
# Create /etc/crypto-policies/policies/modules/COMPAT-OLD-CLIENTS.pmod
sudo update-crypto-policies --set DEFAULT:COMPAT-OLD-CLIENTS

# Option 3: Opt-out specific service
# Configure that service to allow TLS 1.0/1.1

Scenario 3: Testing Before Production

#============================================#
# TEST CRYPTO-POLICY IMPACT
#============================================#

# Current policy
CURRENT=$(update-crypto-policies --show)

# Test with FUTURE policy
sudo update-crypto-policies --set FUTURE
sudo systemctl restart httpd

# Run tests
curl https://localhost/
# Application test suite

# If problems:
sudo update-crypto-policies --set $CURRENT  # Revert
sudo systemctl restart httpd

10.9 Per-Application Overrides

When to Override

Sometimes you need ONE application to opt-out of system policy:

Example: Legacy app needs TLS 1.0, but you want DEFAULT for everything else

#============================================#
# APACHE OVERRIDE (Opt-Out)
#============================================#

# /etc/httpd/conf.d/ssl.conf
# Add this to re-enable TLS 1.0 for Apache only:
SSLProtocol all

# Or use Include to load crypto-policy
Include /etc/crypto-policies/back-ends/httpd.config
# Then override specific settings after

# ⚠️ Note: This opts OUT of crypto-policies for Apache
# You now manage Apache TLS manually again

Better: Use policy modules (see Chapter 23 for details)

10.10 Troubleshooting Crypto-Policy Issues

Common Issues

Issue 1: “no shared cipher”

# Diagnosis
update-crypto-policies --show
# DEFAULT

# Test what ciphers are available
openssl ciphers -v

# Check client request
openssl s_client -connect localhost:443 -cipher 'ALL'

# Solution: Temporarily use LEGACY to identify issue
sudo update-crypto-policies --set LEGACY
# If works → cipher compatibility issue
# Proper fix: Update client or create custom policy

Issue 2: Service fails after policy change

# Symptom
sudo systemctl status httpd
# Failed to start

# Check logs
sudo journalctl -xe -u httpd | grep -i tls

# Revert policy
sudo update-crypto-policies --set DEFAULT
sudo systemctl restart httpd

10.11 Best Practices for RHEL 8

Recommendation: Use DEFAULT Policy

# For most environments:
sudo update-crypto-policies --set DEFAULT

# Reasons:
✅ Balanced security/compatibility
✅ Tested by Red Hat
✅ Meets modern standards
✅ Blocks known weak algorithms
✅ Works with most clients

When to Use Other Policies

Use LEGACY when:

• Temporarily supporting very old clients
• Migration period from RHEL 7
• Testing compatibility
• But: Plan to move back to DEFAULT ASAP!

Use FUTURE when:

• High-security requirements
• All clients are modern
• Want strictest settings
• Planning ahead

Use FIPS when:

• Federal compliance required
• Government contracts
• Regulated industries
• Security certifications needed

10.12 Key Takeaways (RHEL 8)

1. Crypto-policies are THE feature - Learn them well
2. DEFAULT policy is good - Don’t change without reason
3. TLS 1.3 now available - Faster and more secure
4. OpenSSL 1.1.1 - Modern features, better syntax
5. certmonger enhanced - Better automation
6. Migration from RHEL 7 - Test thoroughly
7. Plan for RHEL 9 - OpenSSL 3.x coming

Quick Reference

┌───────────────────────────────────────────────────────────────┐
│ RHEL 8 CRYPTO-POLICIES QUICK REFERENCE                        │
├───────────────────────────────────────────────────────────────┤
│ OpenSSL:        1.1.1k-14                                     │
│ TLS:            1.2, 1.3 (DEFAULT policy)                     │
│ Feature:        System-wide crypto-policies                   │
│                                                               │
│ View policy:    update-crypto-policies --show                 │
│ Set policy:     sudo update-crypto-policies --set <POLICY>    │
│ Policies:       DEFAULT, LEGACY, FUTURE, FIPS                 │
│                                                               │
│ Config files:   /etc/crypto-policies/back-ends/               │
│ Restart:        systemctl restart <services>                  │
│                                                               │
│ Generate key:   openssl genpkey -algorithm RSA -out key.pem   │
│ CSR with SANs:  openssl req -new -addext "subjectAltName=..." │
└───────────────────────────────────────────────────────────────┘

Chapter Navigation

Chapter 11: RHEL 9 Modern Security

Modern Standard: RHEL 9 represents the current state-of-the-art in Linux certificate management with OpenSSL 3.x, enhanced crypto-policies, and stricter security defaults.

11.1 RHEL 9 Overview

Release: May 17, 2022 Support Until: May 31, 2032 Current Version: RHEL 9.8

Major Changes from RHEL 8:

Package: openssl-3.5.5-2.el9_8.x86_64

11.2 OpenSSL 3.5.5 - Major Changes

Provider Architecture (New!)

What Changed: OpenSSL 3.x introduced a “provider” system for different crypto implementations.

#============================================#
# LIST PROVIDERS (RHEL 9)
#============================================#

openssl list -providers

# Output:
# Providers:
#   default
#     name: OpenSSL Default Provider
#     version: 3.5.5
#     status: active
#
#   fips
#     name: OpenSSL FIPS Provider
#     version: 3.5.5
#     status: inactive (unless FIPS mode enabled)
#
#   legacy
#     name: OpenSSL Legacy Provider
#     version: 3.5.5
#     status: inactive
#
#   base
#     name: OpenSSL Base Provider
#     version: 3.5.5
#     status: active

Legacy Algorithms Require Explicit Provider

Breaking Change: MD5, Blowfish, CAST5 need -provider legacy

#============================================#
# USING LEGACY ALGORITHMS (RHEL 9)
#============================================#

# This FAILS on RHEL 9:
openssl md5 file.txt
# Error: unsupported

# This WORKS (explicit provider):
openssl md5 -provider legacy file.txt

# Why: Legacy algorithms disabled by default for security

Modern Key Generation (RHEL 9)

#============================================#
# GENERATE KEYS (RHEL 9)
#============================================#

# RSA 2048 (standard)
openssl genpkey -algorithm RSA -out server.key \
  -pkeyopt rsa_keygen_bits:2048

# RSA 4096 (stronger)
openssl genpkey -algorithm RSA -out server.key \
  -pkeyopt rsa_keygen_bits:4096

# EC P-256 (elliptic curve, recommended)
openssl genpkey -algorithm EC -out ec.key \
  -pkeyopt ec_paramgen_curve:P-256

# EC P-384 (stronger)
openssl genpkey -algorithm EC -out ec.key \
  -pkeyopt ec_paramgen_curve:P-384


#============================================#
# GENERATE CSR WITH SANS (RHEL 9)
#============================================#

openssl req -new -key server.key -out server.csr \
  -subj "/C=US/ST=State/O=Company/CN=server.example.com" \
  -addext "subjectAltName=DNS:server.example.com,DNS:www.example.com,IP:10.0.0.100" \
  -addext "keyUsage=digitalSignature,keyEncipherment" \
  -addext "extendedKeyUsage=serverAuth,clientAuth"

# Verify
openssl req -in server.csr -noout -text | grep -A5 "Subject Alternative Name"

11.3 Enhanced Crypto-Policies (RHEL 9)

Subpolicies (New Feature!)

RHEL 9 introduces policy modifiers:

#============================================#
# CRYPTO-POLICY SUBPOLICIES (RHEL 9)
#============================================#

# Base policy with NO-SHA1 module
sudo update-crypto-policies --set DEFAULT:NO-SHA1

# Multiple modules
sudo update-crypto-policies --set DEFAULT:NO-SHA1:GOST

# Common subpolicies:
# - NO-SHA1: Completely disable SHA-1 (even in signatures)
# - NO-ENFORCE-EMS: Disable Extended Master Secret
# - GOST: Enable GOST algorithms
# - NO-CAMELLIA: Disable Camellia cipher

# View available modules
ls /usr/share/crypto-policies/policies/modules/

Custom Crypto-Policy Modules (RHEL 9)

#============================================#
# CREATE CUSTOM POLICY MODULE
#============================================#

# Create custom module
sudo vi /etc/crypto-policies/policies/modules/CUSTOM.pmod

# Example content:
min_rsa_size = 3072
min_dh_size = 3072
min_dsa_size = 3072

# Apply
sudo update-crypto-policies --set DEFAULT:CUSTOM

# Test
openssl ciphers -v | head

11.4 Stricter Certificate Validation

What’s Stricter in RHEL 9?

#============================================#
# STRICTER VALIDATION EXAMPLES
#============================================#

# 1. SHA-1 signatures completely rejected
openssl verify sha1-signed-cert.crt
# Error: CA md too weak

# 2. Self-signed without proper CA trust rejected
curl https://self-signed.example.com/
# Error: certificate verify failed

# 3. Certificate chain must be complete
# Missing intermediate → connection fails

# 4. Hostname must match (CN or SAN)
openssl s_client -connect server.example.com:443 -servername different.example.com
# Verification error: hostname mismatch

# 5. Keys < 2048 bits rejected
# (even in LEGACY policy, < 1024 rejected)

Impact on Applications

Applications compiled against OpenSSL 3.x:

• May need code changes if using deprecated APIs
• Error handling may be different
• Custom crypto code needs testing

System administrators:

• ✅ Most changes transparent
• ✅ Commands mostly the same
• ⚠️ Stricter validation catches more issues (this is good!)

11.5 RHEL 9 Automation: certmonger, certbot, and IdM ACME

Use the Right Client for the Right CA

#============================================#
# AUTOMATION CHOICES ON RHEL 9
#============================================#

# Native certmonger workflow for FreeIPA / IdM
sudo dnf install certmonger -y
sudo systemctl enable --now certmonger
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/internal.crt \
  -k /etc/pki/tls/private/internal.key \
  -K HTTP/$(hostname -f)@REALM \
  -D $(hostname -f) \
  -C "systemctl reload httpd"

# Public Let's Encrypt workflow
# Use certbot, not a fake certmonger Let's Encrypt CA definition.
sudo certbot certonly --apache -d web.example.com

# IdM ACME workflow (optional)
# This points at your IPA server's ACME directory, not Let's Encrypt.
sudo certbot certonly \
  --server https://ipa.example.com/acme/directory \
  -d host.example.com

Important: IdM ACME and Let’s Encrypt are different CAs. certmonger remains the native RHEL tool for IPA, local CA, and tracked renewal workflows.

11.6 Trust Store Enhancements

Advanced Trust Management

#============================================#
# RHEL 9 TRUST MANAGEMENT
#============================================#

# Add CA (same as RHEL 7/8)
sudo cp corporate-ca.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

# NEW: Purpose-specific trust
trust anchor /path/to/ca.crt --purpose server-auth

# List trust with details
trust list --filter=ca-anchors

# Export specific CA
trust extract --format=pem-bundle --filter=ca-anchors \
  --purpose server-auth /tmp/server-cas.pem

# Remove specific trust
trust anchor --remove "pkcs11:id=%CERT_ID%"

11.7 Common RHEL 9 Issues and Solutions

Issue 1: OpenSSL 3.x API Changes

Problem: Custom application fails with OpenSSL errors

Symptoms:

Error: EVP_PKEY_RSA no longer supported
Error: Provider not available

Solution:

# Check if application is using deprecated APIs
# Application needs recompilation against OpenSSL 3.x

# Temporary: Set compat environment variable (if available)
export OPENSSL_CONF=/etc/pki/tls/openssl-compat.cnf

# Long-term: Update application

Issue 2: SHA-1 Certificates Rejected

Problem: Legacy certificates with SHA-1 signatures fail

Symptoms:

openssl verify cert.crt
# error 3: CA md too weak

Solution:

# Reissue certificate with SHA-256+
# No workaround - SHA-1 is blocked for security

# Check certificate signature
openssl x509 -in cert.crt -noout -text | grep "Signature Algorithm"
# Must show: sha256WithRSAEncryption or better

Issue 3: Legacy Algorithm Not Available

Problem: Application needs MD5/RC4/etc.

Symptoms:

openssl md5 file.txt
# Error: unsupported

Solution:

# Use legacy provider explicitly
openssl md5 -provider legacy file.txt

# For applications: Update to use SHA-256+
# Or configure to load legacy provider

11.8 FIPS Mode on RHEL 9

Improved FIPS Support

#============================================#
# FIPS MODE (RHEL 9)
#============================================#

# Enable FIPS mode
sudo fips-mode-setup --enable
sudo reboot

# Check FIPS status
fips-mode-setup --check
# FIPS mode is enabled.

# Check FIPS provider loaded
openssl list -providers | grep -A3 fips
#   fips
#     name: OpenSSL FIPS Provider
#     version: 3.5.5
#     status: active

# Generate FIPS-compliant certificate
openssl req -new -x509 -days 365 -newkey rsa:2048 \
  -keyout fips.key -out fips.crt \
  -subj "/CN=$(hostname)" -provider fips

RHEL 9 FIPS:

• Uses OpenSSL 3.x FIPS provider
• FIPS 140-2 validated modules
• Transition to FIPS 140-3 in progress

11.9 Migration from RHEL 8

Certificate Impact

Moderate Impact:

• OpenSSL API changes (affects custom apps)
• Stricter validation (catches more issues)
• Legacy algorithms removed
• SHA-1 completely blocked

Pre-Migration Checks

#============================================#
# RHEL 8 → 9 CERTIFICATE PRE-MIGRATION
#============================================#

# 1. Check for SHA-1 certificates (will fail on RHEL 9)
for cert in /etc/pki/tls/certs/*.crt; do
  SIG=$(openssl x509 -in "$cert" -noout -text | grep "Signature Algorithm" | head -2)
  echo "$cert: $SIG"
done | grep -i sha1
# ⚠️ Reissue any SHA-1 certs before migration!

# 2. Check custom applications using OpenSSL
rpm -qa | grep -E "custom|local"
# Test these applications in RHEL 9 environment

# 3. Verify crypto-policy compatibility
update-crypto-policies --show

# 4. Test certificate operations
openssl s_client -connect localhost:443

# 5. Backup everything
tar czf rhel8-certs-backup-$(date +%Y%m%d).tar.gz \
  /etc/pki/tls/ \
  /etc/pki/ca-trust/source/anchors/

11.10 Best Practices for RHEL 9

Recommended Configuration

#============================================#
# RECOMMENDED SETUP (RHEL 9)
#============================================#

# 1. Use DEFAULT crypto-policy (unless specific need)
sudo update-crypto-policies --set DEFAULT

# 2. Use certmonger for native automation
sudo dnf install certmonger
sudo systemctl enable --now certmonger

# 3. For public sites: Use certbot for Let's Encrypt
sudo certbot certonly --apache -d web.example.com

# 4. For internal: Use FreeIPA with certmonger
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/internal.crt \
  -k /etc/pki/tls/private/internal.key \
  -K host/$(hostname -f)@REALM

# 5. Generate EC keys (smaller, faster)
openssl genpkey -algorithm EC -out ec.key \
  -pkeyopt ec_paramgen_curve:P-256

# 6. Always use SANs
openssl req -new -addext "subjectAltName=DNS:..."

11.11 New Features You Should Use

Feature 1: Stronger certmonger + IPA Workflows

# Native RHEL automation for internal certificates
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/internal.crt \
  -k /etc/pki/tls/private/internal.key \
  -K HTTP/$(hostname -f)@REALM \
  -D $(hostname -f) \
  -C "systemctl reload httpd"

# Better status output on RHEL 9
sudo getcert list -v

Feature 2: Enhanced Status Reporting

# More detailed status
sudo getcert list -v

# Better error messages
sudo getcert list -f /etc/pki/tls/certs/web.crt
# Shows exact error reason if renewal fails

Feature 3: Crypto-Policy Subpolicies

# Fine-tune DEFAULT policy
sudo update-crypto-policies --set DEFAULT:NO-SHA1

# Multiple modifiers
sudo update-crypto-policies --set FUTURE:AD-SUPPORT

11.12 Breaking Changes from RHEL 8

API Changes

If you have custom applications:

// RHEL 8 (OpenSSL 1.1.1) - DEPRECATED on RHEL 9:
RSA *rsa = RSA_new();

// RHEL 9 (OpenSSL 3.x) - NEW API:
EVP_PKEY *pkey = EVP_PKEY_new();

Impact: Custom compiled applications may need updates

Command Changes

# Most commands work the same, but some edge cases:

# RHEL 8: This works
openssl md5 file.txt

# RHEL 9: Requires provider
openssl md5 -provider legacy file.txt

# Solution: Use SHA-256 instead
openssl sha256 file.txt

11.13 Common RHEL 9 Scenarios

Scenario 1: Fresh RHEL 9 Apache HTTPS Setup (Internal CA)

#============================================#
# COMPLETE APACHE HTTPS SETUP (RHEL 9)
#============================================#

# 1. Install Apache with mod_ssl
sudo dnf install httpd mod_ssl -y

# 2. Use certmonger + FreeIPA / IdM
sudo dnf install certmonger -y
sudo systemctl enable --now certmonger

# 3. Request certificate
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/web.crt \
  -k /etc/pki/tls/private/web.key \
  -K HTTP/$(hostname -f)@REALM \
  -D $(hostname -f) \
  -C "systemctl reload httpd"

# 4. Wait for certificate (check status)
sudo getcert list

# 5. Configure Apache to use certificate
# /etc/httpd/conf.d/ssl.conf already points to:
#   SSLCertificateFile /etc/pki/tls/certs/localhost.crt
# Update to:
#   SSLCertificateFile /etc/pki/tls/certs/web.crt
#   SSLCertificateKeyFile /etc/pki/tls/private/web.key

# 6. Crypto-policy handles TLS settings automatically!
# No need to set SSLProtocol or SSLCipherSuite

# 7. Open firewall
sudo firewall-cmd --add-service=https --permanent
sudo firewall-cmd --reload

# 8. Start Apache
sudo systemctl enable httpd
sudo systemctl start httpd

# 9. Test
curl -v https://$(hostname -f)/

# 10. Automatic renewal happens ~30 days before expiry!

Result: Fully automated internal HTTPS with FreeIPA and certmonger!

11.14 Troubleshooting RHEL 9 Certificates

Diagnostic Commands

#============================================#
# RHEL 9 CERTIFICATE DIAGNOSTICS
#============================================#

# Check OpenSSL version
openssl version
# OpenSSL 3.5.5

# Check providers
openssl list -providers

# Check crypto-policy
update-crypto-policies --show

# Test connection with TLS 1.3
openssl s_client -connect server:443 -tls1_3

# Check certificate signature algorithm
openssl x509 -in cert.crt -noout -text | grep "Signature Algorithm"
# Must be SHA-256+ on RHEL 9

# Test with legacy provider (if needed)
openssl md5 -provider legacy file.txt

# Check certmonger tracking
sudo getcert list

# View certmonger logs
sudo journalctl -u certmonger -f

Common RHEL 9 Errors

11.15 When to Use RHEL 9

Ideal For:

✅ New deployments - Start with modern security ✅ Security-focused environments - Stricter defaults ✅ Modern applications - Benefit from TLS 1.3 ✅ Long-term support - 10 years maintenance ✅ Compliance requirements - Modern security standards

Migration Timing:

From RHEL 7:

• ✅ Yes! RHEL 7 maintenance ended June 2024
• Plan carefully - big jump (test thoroughly)

From RHEL 8:

• Moderate - OpenSSL 3.x is main change
• Test custom applications first
• SHA-1 certificates must be reissued

11.16 Key Takeaways

1. OpenSSL 3.5.5 provider architecture - Understand providers
2. Stricter validation - Catches security issues (good!)
3. SHA-1 completely blocked - Reissue old certificates
4. Crypto-policy subpolicies - Fine-tune security
5. certmonger stays valuable for IPA and tracked renewal workflows
6. TLS 1.3 mandatory support - Faster, more secure
7. Plan testing - Custom apps may need updates

Quick Reference

┌──────────────────────────────────────────────────────────────┐
│ RHEL 9 CERTIFICATE QUICK REFERENCE                           │
├──────────────────────────────────────────────────────────────┤
│ OpenSSL:        3.5.5 (provider architecture)                │
│ TLS:            1.2, 1.3 (1.0/1.1 removed completely)        │
│ Feature:        Subpolicies, stricter validation             │
│                                                              │
│ Providers:      openssl list -providers                      │
│ Policy:         update-crypto-policies --show                │
│ Subpolicy:      update-crypto-policies --set DEFAULT:NO-SHA1 │
│                                                              │
│ Generate key:   openssl genpkey -algorithm RSA -out key.pem  │
│ EC key:         openssl genpkey -algorithm EC -out ec.pem    │
│                 -pkeyopt ec_paramgen_curve:P-256             │
│                                                              │
│ Public ACME:    certbot certonly --apache -d example.com     │
│ certmonger:     ipa-getcert request ...                     │
│ Legacy algo:    openssl md5 -provider legacy file.txt        │
└──────────────────────────────────────────────────────────────┘

⚠️ SHA-1 is BLOCKED - reissue old certificates!
✅ Use certmonger for automation
✅ DEFAULT policy works for most cases

Chapter Navigation

Chapter 12: RHEL 10 Current Features

Cutting Edge: RHEL 10 GA was released May 20, 2025; RHEL 10.2 is the current minor release. Learn about the latest features and prepare for the future of certificate management on Red Hat Enterprise Linux.

12.1 RHEL 10 Overview

GA Release: May 20, 2025 Current Version: RHEL 10.2 Support Until: May 31, 2035 Status: ✅ Production Release

Key Characteristics:

• OpenSSL Version: 3.5.5-2 (package: openssl-3.5.5-2.el10_2.x86_64)
• Same base as: RHEL 9.8 (OpenSSL 3.5.5)
• Focus: Continued hardening, post-quantum prep, cloud-native
• Philosophy: Incremental improvement over RHEL 9

Important: RHEL 10 features may evolve across minor versions (10.1, 10.2, etc.). Always consult official Red Hat documentation for your specific RHEL 10.x release.

12.2 What’s New vs. RHEL 9?

Key Differences

Package: openssl-3.5.5-2.el10_2.x86_64

Not a Revolutionary Change

Unlike RHEL 7→8 (crypto-policies) or RHEL 8→9 (OpenSSL 3.x), RHEL 10 is an incremental improvement.

Think of it as:

• RHEL 7 → 8: 🚀 Revolutionary (crypto-policies)
• RHEL 8 → 9: 🔄 Major (OpenSSL 3.x)
• RHEL 9 → 10: ⬆️ Incremental (refinements)

12.3 Certificate Management on RHEL 10

Same Foundation as RHEL 9

#============================================#
# RHEL 10 CERTIFICATE BASICS
#============================================#

# Same OpenSSL version as RHEL 9.8
openssl version
# OpenSSL 3.5.5 27 Jan 2026

# Same crypto-policies system
update-crypto-policies --show

# Same certmonger
getcert list

# Same directory structure
ls -la /etc/pki/tls/

Bottom Line: If you know RHEL 9, you know RHEL 10 certificates!

12.4 Enhanced Security Features

Stricter Defaults

#============================================#
# RHEL 10 SECURITY ENHANCEMENTS
#============================================#

# 1. DEFAULT policy is stricter
# - Stronger cipher preferences
# - Additional weak algorithms removed
# - Enhanced validation

# 2. LEGACY policy more restricted
# - Fewer legacy algorithms allowed
# - Stronger minimums even in LEGACY

# 3. Container certificate management improved
# - Better integration with Podman
# - Simplified cert mounting
# - Enhanced secret management

Post-Quantum Cryptography Preparation

Foundation for Future:

# RHEL 10 prepares for post-quantum algorithms
# (Not yet default, but infrastructure ready)

# Future capability (as standards finalize):
# - ML-KEM (Module-Lattice Key Encapsulation)
# - ML-DSA (Module-Lattice Digital Signatures)
# - Hybrid classical/quantum cryptography

# Current status: Monitoring NIST standards
# Expected: RHEL 10.x minor releases will add PQC support

Note: Post-quantum cryptography is still evolving. RHEL 10 provides foundation, actual implementation will come as standards are finalized.

12.5 RHEL 10-Specific Features

Feature 1: Enhanced Crypto-Policy Modules

#============================================#
# RHEL 10 CRYPTO-POLICY IMPROVEMENTS
#============================================#

# More granular control
sudo update-crypto-policies --set DEFAULT:NO-SHA1

# Better validation
update-crypto-policies --check

# Improved error messages when policies conflict

Feature 2: Improved Container Certificate Support

#============================================#
# CONTAINERS WITH CERTIFICATES (RHEL 10)
#============================================#

# Easier certificate mounting in Podman
podman run -d \
  -v /etc/pki/tls/certs/web.crt:/certs/web.crt:ro \
  -v /etc/pki/tls/private/web.key:/certs/web.key:ro \
  -p 443:443 \
  nginx

# Enhanced secret management
podman secret create web-cert /etc/pki/tls/certs/web.crt
podman secret create web-key /etc/pki/tls/private/web.key

# Use secrets in container
podman run -d --secret web-cert --secret web-key nginx

Feature 3: Enhanced FIPS Mode

#============================================#
# FIPS ON RHEL 10
#============================================#

# FIPS mode with OpenSSL 3.x FIPS provider
sudo fips-mode-setup --enable
sudo reboot

# Check FIPS status
fips-mode-setup --check

# RHEL 10: Transition toward FIPS 140-3
# Current: Still FIPS 140-2 validated modules
# Future: FIPS 140-3 compliance as certification completes

12.6 Migration from RHEL 9

Should You Upgrade?

Upgrade Considerations:

Reasons to Upgrade:

• ✅ Want 10+ years of support (until 2035)
• ✅ Need latest security enhancements
• ✅ Future-proofing (post-quantum prep)
• ✅ Enhanced container support
• ✅ Latest features and improvements

Reasons to Wait:

• ⏸️ RHEL 9 supported until 2032
• ⏸️ No urgent certificate-related features
• ⏸️ Let others test RHEL 10 in production first
• ⏸️ Want to wait for RHEL 10.3 or 10.4

Certificate Impact: LOW

• Same OpenSSL base (3.5.5)
• Same tools and commands
• Minimal breaking changes
• Mostly transparent

Migration Process

#============================================#
# RHEL 9 → RHEL 10 CERTIFICATE MIGRATION
#============================================#

# 1. Pre-migration: Verify certificates
for cert in /etc/pki/tls/certs/*.crt; do
  openssl x509 -in "$cert" -noout -text | grep "Signature Algorithm"
done
# All should show SHA-256+ (no SHA-1 or MD5)

# 2. Backup
tar czf rhel9-certs-backup-$(date +%Y%m%d).tar.gz \
  /etc/pki/tls/ \
  /etc/pki/ca-trust/source/anchors/

# 3. Perform RHEL upgrade
sudo leapp upgrade

# 4. Verify crypto-policy
update-crypto-policies --show

# 5. Restart services
sudo systemctl restart httpd nginx postfix

# 6. Test certificates
curl -v https://localhost/
openssl s_client -connect localhost:443

# 7. Check certmonger
sudo getcert list

12.7 Best Practices for RHEL 10

Recommended Setup

#============================================#
# RHEL 10 RECOMMENDED CONFIGURATION
#============================================#

# 1. Use DEFAULT crypto-policy (already optimal)
sudo update-crypto-policies --set DEFAULT

# 2. Prefer TLS 1.3
# (Automatically preferred by DEFAULT policy)

# 3. Use EC keys for new certificates
openssl genpkey -algorithm EC -out ec.key \
  -pkeyopt ec_paramgen_curve:P-256

# 4. Automate with the right tool
# Public Let's Encrypt certificate: use certbot
sudo certbot certonly --apache -d web.example.com

# Internal FreeIPA / IdM certificate: use certmonger
# sudo ipa-getcert request \
#   -f /etc/pki/tls/certs/web.crt \
#   -k /etc/pki/tls/private/web.key \
#   -K HTTP/web.example.com@REALM \
#   -D web.example.com \
#   -C "systemctl reload httpd"

# 5. Monitor certificates
# Use built-in monitoring or external tools

# 6. Plan for future PQC
# Keep up with RHEL 10.x minor releases

12.8 Looking Ahead: Post-Quantum Readiness

What is Post-Quantum Cryptography?

Problem: Future quantum computers could break current encryption (RSA, ECC) Solution: New quantum-resistant algorithms

NIST Standards (Finalized 2024):

• ML-KEM-768 (Key Encapsulation)
• ML-DSA-65 (Digital Signatures)
• SLH-DSA (Stateless signatures)

RHEL 10 Role:

• Provides foundation for PQC
• OpenSSL 3.x architecture supports new algorithms
• Future RHEL 10.x releases will add PQC support

Hybrid Cryptography (Future)

# Future capability in RHEL 10.x:
# Use both classical AND quantum-resistant crypto

# Example (conceptual - not yet in RHEL 10.2):
openssl genpkey -algorithm hybrid-rsa-mlkem768 -out hybrid.key

# Provides:
# - Security against classical attacks (RSA)
# - Security against quantum attacks (ML-KEM)

Note: PQC support is coming in future RHEL 10.x minor versions as standards are finalized and tested.

12.9 What Stays the Same

No Major Breaking Changes

#============================================#
# FAMILIAR COMMANDS STILL WORK
#============================================#

# Generate key (same as RHEL 9)
openssl genpkey -algorithm RSA -out server.key \
  -pkeyopt rsa_keygen_bits:2048

# Generate CSR (same as RHEL 9)
openssl req -new -key server.key -out server.csr \
  -subj "/CN=server.example.com" \
  -addext "subjectAltName=DNS:server.example.com"

# View certificate (same)
openssl x509 -in cert.crt -noout -text

# Test connection (same)
openssl s_client -connect server:443

# Trust management (same)
sudo cp ca.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

# certmonger (same)
sudo getcert list

# crypto-policies (same)
update-crypto-policies --show

If you know RHEL 9, you’re ready for RHEL 10!

12.10 When to Adopt RHEL 10

Adoption Timeline Recommendations

Early Adopters (2025-2026):

• Testing environments
• Non-critical workloads
• Want latest features
• Security research

Mainstream (2026-2027):

• New deployments
• Refreshed infrastructure
• After RHEL 10.3/10.4 release
• When major apps certified

Conservative (2027-2028):

• Critical production systems
• Stable workloads
• After extensive community testing
• When migration from RHEL 9 necessary

Current Recommendation (Late 2025):

• ✅ New projects: Consider RHEL 10
• ⏸️ Existing RHEL 9: No urgency to upgrade
• ✅ RHEL 8 or older: Evaluate RHEL 9 or 10
• ❌ RHEL 7: Upgrade required (support ended)

12.11 Monitoring RHEL 10 Evolution

Stay Updated

# Check RHEL 10 minor version
cat /etc/redhat-release
# Red Hat Enterprise Linux release 10.2 (Coughlan)

# Check for updates
sudo dnf check-update

# Monitor Red Hat announcements
# - https://access.redhat.com/articles/3078
# - RHEL 10 release notes
# - Red Hat security advisories

# Subscribe to Red Hat newsletters
# Follow release notes for 10.3, 10.4, etc.

Features to Watch For

Expected in RHEL 10.x minor releases:

• Post-quantum cryptography support
• Additional crypto-policy enhancements
• Further container integration
• Enhanced automation tools
• Additional FIPS 140-3 modules

12.12 Practical RHEL 10 Certificate Setup

Complete Example: Modern HTTPS Setup

#!/bin/bash
# Complete modern HTTPS setup on RHEL 10

echo "=== RHEL 10 Modern HTTPS Setup ==="

# 1. Install packages
sudo dnf install -y httpd mod_ssl epel-release certbot python3-certbot-apache

# 2. Enable services
sudo systemctl enable --now httpd

# 3. Request Let's Encrypt certificate with certbot
sudo certbot --apache -d $(hostname -f)

# 4. Verify the certificate
sudo certbot certificates

# 5. Update Apache configuration
# certbot usually updates Apache automatically; adjust manually only if needed
sudo sed -i "s|SSLCertificateFile.*|SSLCertificateFile /etc/letsencrypt/live/$(hostname -f)/fullchain.pem|" \
  /etc/httpd/conf.d/ssl.conf
sudo sed -i "s|SSLCertificateKeyFile.*|SSLCertificateKeyFile /etc/letsencrypt/live/$(hostname -f)/privkey.pem|" \
  /etc/httpd/conf.d/ssl.conf

# 6. Crypto-policy already optimal (DEFAULT)
update-crypto-policies --show

# 7. Open firewall
sudo firewall-cmd --add-service=https --permanent
sudo firewall-cmd --reload

# 8. Reload Apache
sudo systemctl reload httpd

# 9. Test
curl -v https://$(hostname -f)/

echo "✅ RHEL 10 modern HTTPS setup complete!"
echo "   - TLS 1.3 supported"
echo "   - Let's Encrypt certificate"
echo "   - Automatic renewal enabled"
echo "   - Optimal security (DEFAULT policy)"

12.13 Future-Proofing Strategies

Preparing for RHEL 10.x Evolution

#============================================#
# FUTURE-PROOF CERTIFICATE MANAGEMENT
#============================================#

# 1. Use modern algorithms (ready for PQC transition)
# Prefer EC over RSA
openssl genpkey -algorithm EC -out ec.key -pkeyopt ec_paramgen_curve:P-256

# 2. Keep certificates short-lived (90 days or less)
# Easier to rotate when algorithms change

# 3. Automate everything
# Use certbot for public ACME and certmonger for IPA/internal workflows

# 4. Monitor Red Hat announcements
# Subscribe to security and release notifications

# 5. Test PQC when available
# Be early tester of new features in RHEL 10.x

# 6. Document your setup
# Makes future transitions easier

12.14 Known Issues and Workarounds

Issue 1: Same as RHEL 9 (OpenSSL 3.x)

Most RHEL 9 issues apply to RHEL 10:

• Legacy algorithms need -provider legacy
• SHA-1 blocked
• Custom apps may need OpenSSL 3.x updates

Reference: See Chapter 11 for OpenSSL 3.x issues

Issue 2: Even Stricter Validation

RHEL 10 may catch issues RHEL 9 allowed:

# Example: Marginal certificate that worked on RHEL 9
# might fail on RHEL 10

# Solution: Always use best practices
# - SHA-256+ signatures
# - 2048+ bit keys (4096 recommended)
# - Proper SANs
# - Valid trust chains

12.15 When to Choose RHEL 10

Decision Matrix

12.16 Key Takeaways

1. RHEL 10 = RHEL 9 + incremental improvements
2. Same OpenSSL 3.5.5 base - No major API changes
3. Stricter security defaults - Good for security
4. Post-quantum preparation - Future-ready infrastructure
5. No urgent certificate changes - Transition is smooth
6. RHEL 9 knowledge transfers - Same tools and commands
7. Watch for minor releases - 10.3, 10.4 may add features

12.17 Troubleshooting RHEL 10

Diagnostic Approach

#============================================#
# RHEL 10 CERTIFICATE TROUBLESHOOTING
#============================================#

# Use troubleshooting methodology (Chapter 27) and RHEL 9 patterns (Chapter 11)

# 1. Verify RHEL 10 version
cat /etc/redhat-release
# Red Hat Enterprise Linux release 10.2 (Coughlan)

# 2. Check OpenSSL
openssl version
# OpenSSL 3.5.5

# 3. Check crypto-policy
update-crypto-policies --show

# 4. Test certificate
openssl x509 -in cert.crt -noout -text

# 5. Test connection
openssl s_client -connect server:443 -tls1_3

# 6. Check providers (if issues)
openssl list -providers

# 7. Check logs
sudo journalctl -xe | grep -i cert

No new troubleshooting techniques needed - same as RHEL 9!

12.18 Recommended Migration Path

From RHEL 9 to RHEL 10

#============================================#
# CERTIFICATE-SAFE RHEL 9→10 MIGRATION
#============================================#

# Phase 1: Preparation
# - Backup all certificates
# - Document current configuration
# - Test in lab environment

# Phase 2: Migration
# - Use standard RHEL upgrade process
# - Certificates should transfer seamlessly

# Phase 3: Verification
# - Verify crypto-policy unchanged
# - Test all certificate operations
# - Confirm certmonger tracking maintained
# - Test services using certificates

# Phase 4: Optimization
# - Consider EC keys for new certificates
# - Review and update crypto-policy if needed
# - Monitor for RHEL 10.x enhancements

12.19 Documentation and Resources

Official Resources

## RHEL 10 Certificate Resources

### Official Documentation
- RHEL 10 Release Notes (check for your specific 10.x version)
- https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/10

### Security Updates
- https://access.redhat.com/security/
- Subscribe to RHEL security announcements

### Crypto-Policies
- https://access.redhat.com/articles/3642912
- Check for RHEL 10-specific updates

### Support
- Red Hat Customer Portal
- Red Hat Support Cases
- RHEL community forums

12.20 Quick Reference

┌──────────────────────────────────────────────────────────────┐
│ RHEL 10 CERTIFICATE QUICK REFERENCE                          │
├──────────────────────────────────────────────────────────────┤
│ OpenSSL:      3.5.5-2 (same as RHEL 9.8)                     │
│ TLS:          1.3 preferred, 1.2 supported                   │
│ GA Release:   May 20, 2025 (RHEL 10.0)                       │
│ Status:       Production ready                               │
│                                                              │
│ Key Change:   Incremental security improvements              │
│ Migration:    Low impact from RHEL 9                         │
│ Commands:     Same as RHEL 9                                 │
│ Tools:        Same as RHEL 9                                 │
│                                                              │
│ Future:       Post-quantum crypto preparation                │
│               Watch for features in 10.3, 10.4+              │
│                                                              │
│ Verify:       cat /etc/redhat-release                        │
│               openssl version                                │
│               update-crypto-policies --show                  │
└──────────────────────────────────────────────────────────────┘

✅ If you know RHEL 9 certificates, you know RHEL 10!
⚠️ Always check official docs for your specific 10.x minor version

Chapter Navigation

Chapter 13: Cross-Version Compatibility

Real-World Challenge: Your environment probably has RHEL 7, 8, and 9 systems all talking to each other. How do you make certificates work across all versions?

13.1 The Mixed Environment Reality

Most enterprises don’t upgrade everything at once. You’ll encounter:

Production Environment (Typical):
├── Legacy App Server (RHEL 7)
├── Database Server (RHEL 8)
├── Web Tier (RHEL 9)
├── Management Node (RHEL 10)
└── Clients: Windows, Mac, Linux, Mobile

Challenge: These systems have different:

• TLS version support
• Cipher suites
• Certificate validation rules
• OpenSSL versions
• Crypto-policies (or lack thereof)

13.2 Common Compatibility Issues

Issue 1: TLS Version Mismatches

Scenario: RHEL 9 server (TLS 1.2+ only) ← RHEL 7 client (TLS 1.0/1.1 default)

# RHEL 7 client trying to connect to RHEL 9 server
curl https://rhel9-server.example.com/
# Error: SSL routines:ssl3_get_record:wrong version number

# Why: RHEL 7 tries TLS 1.0 first, RHEL 9 rejects it

Solution:

# Option 1: Update RHEL 7 client to use TLS 1.2
# Edit /etc/httpd/conf.d/ssl.conf (if Apache client)
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1

# Option 2: Temporary LEGACY policy on RHEL 9 (NOT recommended)
# sudo update-crypto-policies --set LEGACY  # On RHEL 9 server

# Option 3: Best - Upgrade RHEL 7 systems!

Issue 2: Cipher Suite Mismatches

Scenario: Modern server doesn’t support old client ciphers

# RHEL 7 client → RHEL 9 server
openssl s_client -connect rhel9-server:443 -cipher '3DES'
# Error: no shared cipher

Why: 3DES is blocked in RHEL 8+ DEFAULT policy

Solution:

# Check what ciphers are available
openssl ciphers -v 'HIGH:!aNULL:!MD5' | head

# Test specific cipher on client
openssl s_client -connect server:443 -cipher 'AES256-GCM-SHA384'

# On RHEL 9 server, if you MUST support old clients:
sudo update-crypto-policies --set LEGACY  # Temporary!

Issue 3: Certificate Validation Differences

Scenario: SHA-1 signed certificate

Certificate with SHA-1 signature:
├── ✅ Works on RHEL 7
├── ❌ Rejected by RHEL 8 DEFAULT
├── ❌ Rejected by RHEL 9
└── ❌ Rejected by RHEL 10

Solution:

# Reissue certificate with SHA-256 or better
openssl req -new -key server.key -out server.csr -sha256

# Verify signature algorithm
openssl x509 -in cert.crt -noout -text | grep "Signature Algorithm"
# Should show: sha256WithRSAEncryption (or better)

13.3 Compatibility Matrix

Client → Server Compatibility

Legend:

• ✅ Works without changes
• ⚠️ Works with configuration changes
• ❌ Incompatible without major updates

13.4 Certificate Requirements for Maximum Compatibility

The “Universal” Certificate Profile

To work across all RHEL versions (7-10) and external clients:

# Certificate Requirements:
✅ RSA key: 2048 bits minimum (4096 for future-proofing)
✅ Signature: SHA-256 or better (not SHA-1!)
✅ Subject Alternative Names (SANs) required
✅ Validity: ≤ 365 days (browser requirement)
✅ Key Usage: Proper extensions set

❌ Avoid: SHA-1 signatures
❌ Avoid: RSA < 2048 bits
❌ Avoid: Missing SANs
❌ Avoid: CN-only certificates

Generate a Compatible Certificate

#============================================#
# STEP 1: Generate Key (works on all RHEL versions)
#============================================#

# RSA 2048 (minimum, compatible)
openssl genpkey -algorithm RSA -out universal.key -pkeyopt rsa_keygen_bits:2048

# Or RSA 4096 (better, still compatible)
openssl genpkey -algorithm RSA -out universal.key -pkeyopt rsa_keygen_bits:4096


#============================================#
# STEP 2: Create CSR with SANs
#============================================#

openssl req -new -key universal.key -out universal.csr \
  -subj "/C=US/ST=State/L=City/O=Company/CN=server.example.com" \
  -addext "subjectAltName=DNS:server.example.com,DNS:www.example.com,IP:10.0.0.100" \
  -addext "keyUsage=digitalSignature,keyEncipherment" \
  -addext "extendedKeyUsage=serverAuth,clientAuth"


#============================================#
# STEP 3: Verify CSR
#============================================#

openssl req -in universal.csr -noout -text | grep -A2 "Subject Alternative Name"
# Should show your SANs

openssl req -in universal.csr -noout -text | grep "Public-Key"
# Should show: Public-Key: (2048 bit) or higher

13.5 Testing Cross-Version Compatibility

Test Suite Script

#!/bin/bash
# test-cert-compatibility.sh
# Tests if certificate works from various RHEL versions

SERVER_HOST="server.example.com"
SERVER_PORT="443"
CERT_FILE="/etc/pki/tls/certs/server.crt"

echo "=== Certificate Compatibility Test Suite ==="
echo ""

#============================================#
# TEST 1: Certificate Properties
#============================================#

echo "1. Certificate Properties:"
echo "   Signature Algorithm:"
openssl x509 -in "$CERT_FILE" -noout -text | grep "Signature Algorithm" | head -1

echo "   Key Size:"
openssl x509 -in "$CERT_FILE" -noout -text | grep "Public-Key"

echo "   SANs:"
openssl x509 -in "$CERT_FILE" -noout -ext subjectAltName 2>/dev/null || echo "   No SANs found!"

echo ""


#============================================#
# TEST 2: TLS Version Support
#============================================#

echo "2. TLS Version Support:"

for version in tls1 tls1_1 tls1_2 tls1_3; do
  if openssl s_client -connect "$SERVER_HOST:$SERVER_PORT" -"$version" </dev/null 2>&1 | grep -q "Cipher"; then
    echo "   ${version//_/.}: ✅ Supported"
  else
    echo "   ${version//_/.}: ❌ Not supported"
  fi
done

echo ""


#============================================#
# TEST 3: Cipher Suite Compatibility
#============================================#

echo "3. Common Cipher Tests:"

# Modern cipher (RHEL 8+)
if openssl s_client -connect "$SERVER_HOST:$SERVER_PORT" -cipher 'ECDHE-RSA-AES256-GCM-SHA384' </dev/null 2>&1 | grep -q "Cipher"; then
  echo "   Modern cipher (ECDHE-RSA-AES256-GCM-SHA384): ✅"
else
  echo "   Modern cipher: ❌"
fi

# Legacy cipher (RHEL 7)
if openssl s_client -connect "$SERVER_HOST:$SERVER_PORT" -cipher 'AES256-SHA' </dev/null 2>&1 | grep -q "Cipher"; then
  echo "   Legacy cipher (AES256-SHA): ✅ (may indicate LEGACY policy)"
else
  echo "   Legacy cipher: ❌ (good for security)"
fi


#============================================#
# TEST 4: Certificate Validation
#============================================#

echo ""
echo "4. Certificate Validation:"

if openssl verify -CAfile /etc/pki/tls/certs/ca-bundle.crt "$CERT_FILE" | grep -q "OK"; then
  echo "   Trust chain: ✅ Valid"
else
  echo "   Trust chain: ❌ Invalid"
fi

echo ""
echo "=== Test Complete ==="

Usage:

chmod +x test-cert-compatibility.sh
sudo ./test-cert-compatibility.sh

13.6 Handling Specific Compatibility Scenarios

Scenario 1: RHEL 7 Client → RHEL 9 Server

Problem: Connection fails with TLS version error

Client-Side Fix (RHEL 7):

# For curl
curl --tlsv1.2 https://rhel9-server/

# For wget
wget --secure-protocol=TLSv1_2 https://rhel9-server/

# For applications using OpenSSL, set environment variable
export OPENSSL_CONF=/etc/pki/tls/openssl-tls12.cnf

# Create custom config
cat > /etc/pki/tls/openssl-tls12.cnf << 'EOF'
openssl_conf = default_conf

[default_conf]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
MinProtocol = TLSv1.2
CipherString = DEFAULT@SECLEVEL=1
EOF

Server-Side Fix (RHEL 9) - NOT RECOMMENDED:

# Only if absolutely necessary and temporarily!
sudo update-crypto-policies --set LEGACY
sudo systemctl restart httpd  # Or your service

Scenario 2: Mixed CA Trust

Problem: Corporate CA trusted on some systems but not others

Solution: Consistent trust store across all versions

#============================================#
# DEPLOYMENT SCRIPT (run on all RHEL versions)
#============================================#

#!/bin/bash
# deploy-corporate-ca.sh

CA_CERT_URL="http://pki.example.com/ca-chain.crt"
CA_CERT_FILE="/etc/pki/ca-trust/source/anchors/corporate-ca-chain.crt"

# Download CA certificate
curl -o "$CA_CERT_FILE" "$CA_CERT_URL"

# Update trust store (works on all RHEL versions)
update-ca-trust extract

# Verify
if trust list | grep -q "Corporate Root CA"; then
  echo "✅ Corporate CA installed successfully"
else
  echo "❌ Corporate CA installation failed"
  exit 1
fi

Scenario 3: Application Using Old TLS Library

Problem: Java 7 application can’t connect to modern servers

Check Java TLS Support:

# Check Java version
java -version

# Test TLS support
java -Djavax.net.debug=ssl:handshake -jar app.jar 2>&1 | grep "TLS"

Options:

# Option 1: Upgrade Java (best)
sudo dnf install java-11-openjdk

# Option 2: Enable TLS 1.2 in Java 7 (if upgrade impossible)
# Add to Java startup:
-Dhttps.protocols=TLSv1.2

# Option 3: Use wrapper script
#!/bin/bash
export JAVA_OPTS="-Dhttps.protocols=TLSv1.2 -Djavax.net.ssl.trustStore=/etc/pki/java/cacerts"
java $JAVA_OPTS -jar /path/to/app.jar

13.7 Crypto-Policy Compatibility

Understanding Policy Impact Across Versions

#============================================#
# RHEL 7 (No crypto-policies)
#============================================#

# Manual configuration in each app
# Apache: /etc/httpd/conf.d/ssl.conf
# NGINX: /etc/nginx/nginx.conf
# Postfix: /etc/postfix/main.cf


#============================================#
# RHEL 8/9/10 (crypto-policies)
#============================================#

# System-wide control
update-crypto-policies --set DEFAULT

# To support RHEL 7 clients, might need:
update-crypto-policies --set LEGACY  # Temporarily!

Policy Equivalents for Mixed Environments

If you need to maintain compatibility:

Option A: Use LEGACY on modern systems (not recommended long-term)

# On RHEL 8/9/10 servers
sudo update-crypto-policies --set LEGACY

Option B: Configure RHEL 7 to match DEFAULT (recommended)

# On RHEL 7, manually configure to match RHEL 8+ DEFAULT
# Apache example:
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite HIGH:!aNULL:!MD5:!3DES:!CAMELLIA
SSLHonorCipherOrder on

13.8 Migration Path: Gradual Upgrade

Phase 1: Prepare RHEL 7 (Pre-Migration)

# 1. Reissue all certificates with SHA-256+
# 2. Test TLS 1.2 compatibility
# 3. Update cipher configurations
# 4. Document current certificate inventory

Phase 2: Deploy RHEL 8 (Transition)

# 1. Start with LEGACY policy
sudo update-crypto-policies --set LEGACY

# 2. Deploy services
# 3. Test thoroughly
# 4. Gradually switch to DEFAULT
sudo update-crypto-policies --set DEFAULT

Phase 3: Upgrade to RHEL 9 (Modernization)

# 1. All clients should be RHEL 8+ or TLS 1.2 capable
# 2. Use DEFAULT policy
# 3. Monitor for compatibility issues
# 4. Consider FUTURE policy after stabilization

13.9 Troubleshooting Cross-Version Issues

Diagnostic Commands

#============================================#
# ON CLIENT
#============================================#

# Test specific TLS version
openssl s_client -connect server:443 -tls1_2

# Test with verbose output
curl -v --tlsv1.2 https://server/

# Check client OpenSSL
openssl version
openssl ciphers -v


#============================================#
# ON SERVER
#============================================#

# Check crypto-policy (RHEL 8+)
update-crypto-policies --show

# Check OpenSSL config
openssl version
cat /etc/crypto-policies/back-ends/opensslcnf.config

# Test server certificate
openssl s_client -connect localhost:443 -servername $(hostname)

# Check service logs
sudo journalctl -xe | grep -i tls
sudo tail -f /var/log/httpd/ssl_error_log

Common Error Messages

13.10 Best Practices for Mixed Environments

1. Standardize Certificate Issuance

# Certificate Standard (example)
Algorithm: RSA
Key Size: 2048 bits minimum (4096 preferred)
Signature: SHA-256 or better
Validity: 365 days maximum
SANs: Always include
Extensions: Proper key usage set

2. Maintain Consistent Trust Stores

# Deploy script for all systems
for host in rhel7-hosts rhel8-hosts rhel9-hosts; do
  ssh "$host" 'sudo cp /path/to/ca.crt /etc/pki/ca-trust/source/anchors/ && sudo update-ca-trust'
done

3. Test Before Deploying

# Test matrix
RHEL 7 client → RHEL 7 server ✓
RHEL 7 client → RHEL 8 server ✓
RHEL 7 client → RHEL 9 server ✓
RHEL 8 client → RHEL 7 server ✓
RHEL 8 client → RHEL 9 server ✓
RHEL 9 client → RHEL 8 server ✓

4. Document Your Environment

## Certificate Compatibility Matrix

### Servers:
- App Server 1: RHEL 7.9, TLS 1.0-1.2, RSA 2048
- Database: RHEL 8.10, DEFAULT policy, RSA 2048
- Web Tier: RHEL 9.8, DEFAULT policy, RSA 4096

### Known Limitations:
- RHEL 7 systems require TLS 1.0/1.1 for legacy app X
- Database requires specific cipher: AES256-GCM-SHA384

### Upgrade Plan:
- Q1 2025: Migrate App Server 1 to RHEL 8
- Q2 2025: Update all certificates to RSA 4096

13.11 Key Takeaways

1. Mixed environments are normal - Plan for compatibility
2. TLS 1.2+ is the minimum for modern systems
3. SHA-256+ signatures required for RHEL 8+
4. Crypto-policies changed everything (RHEL 8+)
5. Test across all versions before deploying
6. Document everything - especially exceptions
7. Upgrade path is gradual - don’t rush, test thoroughly

Quick Reference

┌──────────────────────────────────────────────────────────────┐
│ CROSS-VERSION COMPATIBILITY CHECKLIST                        │
├──────────────────────────────────────────────────────────────┤
│ ✅ RSA 2048+ bits                                            │
│ ✅ SHA-256+ signature                                        │
│ ✅ SANs included                                             │
│ ✅ TLS 1.2+ support                                          │
│ ✅ Modern ciphers                                            │
│ ✅ Consistent CA trust                                       │
│ ✅ Tested across all versions                                │
└──────────────────────────────────────────────────────────────┘

Test command:
openssl s_client -connect server:443 -tls1_2 -servername server

Check policy (RHEL 8+):
update-crypto-policies --show

Chapter Navigation

Chapter 14: Apache httpd on RHEL

Most Common: Apache (httpd) is the most widely deployed web server on RHEL. Master Apache HTTPS configuration across all RHEL versions.

14.1 Apache on RHEL Overview

Package Name: httpd SSL/TLS Module: mod_ssl Config Location: /etc/httpd/conf.d/ssl.conf Certificate Path: /etc/pki/tls/certs/ Key Path: /etc/pki/tls/private/

Version Comparison

14.2 Installation

RHEL 7

#============================================#
# INSTALL APACHE WITH SSL (RHEL 7)
#============================================#

sudo yum install httpd mod_ssl -y
sudo systemctl enable httpd
sudo systemctl start httpd

# Open firewall
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

# Verify
systemctl status httpd

RHEL 8/9/10

#============================================#
# INSTALL APACHE WITH SSL (RHEL 8/9/10)
#============================================#

sudo dnf install httpd mod_ssl -y
sudo systemctl enable httpd
sudo systemctl start httpd

# Open firewall
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

# Verify
systemctl status httpd
ss -tlnp | grep :443  # Check if listening on 443

14.3 Basic SSL Configuration

Default SSL Configuration

# Main SSL config file
/etc/httpd/conf.d/ssl.conf

# Key directives:
SSLEngine on
SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

Complete Virtual Host Example

#============================================#
# /etc/httpd/conf.d/ssl.conf
# Or /etc/httpd/conf.d/mysite-ssl.conf
#============================================#

<VirtualHost *:443>
    ServerName www.example.com
    ServerAlias example.com
    DocumentRoot /var/www/html

    # Enable SSL/TLS
    SSLEngine on

    # Certificate files
    SSLCertificateFile      /etc/pki/tls/certs/www.example.com.crt
    SSLCertificateKeyFile   /etc/pki/tls/private/www.example.com.key
    SSLCertificateChainFile /etc/pki/tls/certs/chain.crt

    # TLS protocols (RHEL 7 - manual config)
    SSLProtocol             all -SSLv3 -TLSv1 -TLSv1.1

    # Cipher suite (RHEL 7 - manual)
    SSLCipherSuite          HIGH:!aNULL:!MD5:!3DES:!RC4
    SSLHonorCipherOrder     on

    # HSTS (recommended)
    Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"

    # Logging
    ErrorLog  /var/log/httpd/ssl_error_log
    CustomLog /var/log/httpd/ssl_access_log combined
</VirtualHost>

14.4 Version-Specific Configuration

RHEL 7: Manual SSL Configuration

#============================================#
# APACHE SSL - RHEL 7 BEST PRACTICES
#============================================#

<VirtualHost *:443>
    ServerName www.example.com
    SSLEngine on

    # Certificates
    SSLCertificateFile      /etc/pki/tls/certs/www.crt
    SSLCertificateKeyFile   /etc/pki/tls/private/www.key
    SSLCertificateChainFile /etc/pki/tls/certs/chain.crt

    # REQUIRED: Disable old TLS versions manually
    SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1

    # REQUIRED: Strong ciphers only
    SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256
    SSLHonorCipherOrder on

    # Security headers
    Header always set Strict-Transport-Security "max-age=31536000"
    Header always set X-Frame-Options DENY
    Header always set X-Content-Type-Options nosniff
</VirtualHost>

RHEL 8/9/10: Crypto-Policies Integrated

#============================================#
# APACHE SSL - RHEL 8/9/10 WITH CRYPTO-POLICIES
#============================================#

<VirtualHost *:443>
    ServerName www.example.com
    SSLEngine on

    # Certificates
    SSLCertificateFile      /etc/pki/tls/certs/www.crt
    SSLCertificateKeyFile   /etc/pki/tls/private/www.key
    SSLCertificateChainFile /etc/pki/tls/certs/chain.crt

    # NO NEED to set SSLProtocol or SSLCipherSuite!
    # Crypto-policies handles it automatically
    # (unless you have specific requirements)

    # Security headers (still manual)
    Header always set Strict-Transport-Security "max-age=31536000"
    Header always set X-Frame-Options DENY
    Header always set X-Content-Type-Options nosniff
</VirtualHost>

Key Difference: On RHEL 8+, crypto-policies automatically configures TLS versions and ciphers!

Checking Crypto-Policy Integration

#============================================#
# VERIFY CRYPTO-POLICY (RHEL 8/9/10)
#============================================#

# Check current policy
update-crypto-policies --show

# View Apache-specific policy
cat /etc/crypto-policies/back-ends/httpd.config

# Apache automatically includes this
grep -r "crypto-policies" /etc/httpd/

14.5 Certificate Generation for Apache

Complete Workflow

#============================================#
# GENERATE CERTIFICATE FOR APACHE (ALL VERSIONS)
#============================================#

# Step 1: Generate private key
sudo openssl genpkey -algorithm RSA \
  -out /etc/pki/tls/private/www.example.com.key \
  -pkeyopt rsa_keygen_bits:2048

# Step 2: Set permissions
sudo chmod 600 /etc/pki/tls/private/www.example.com.key
sudo chown root:root /etc/pki/tls/private/www.example.com.key

# Step 3: Generate CSR with SANs
sudo openssl req -new \
  -key /etc/pki/tls/private/www.example.com.key \
  -out /tmp/www.example.com.csr \
  -subj "/C=US/ST=State/L=City/O=Company/CN=www.example.com" \
  -addext "subjectAltName=DNS:www.example.com,DNS:example.com"

# Step 4: Submit CSR to CA, receive certificate

# Step 5: Install certificate
sudo cp www.example.com.crt /etc/pki/tls/certs/
sudo chmod 644 /etc/pki/tls/certs/www.example.com.crt

# Step 6: If using intermediate certificates, install chain
sudo cp chain.crt /etc/pki/tls/certs/www.example.com-chain.crt

# Step 7: Update Apache config (see section 13.3)

# Step 8: Test configuration
sudo apachectl configtest

# Step 9: Reload Apache
sudo systemctl reload httpd

# Step 10: Test HTTPS
curl -v https://www.example.com/
openssl s_client -connect www.example.com:443 -servername www.example.com

14.6 certmonger Integration (Automation!)

Using certmonger with Apache

#============================================#
# AUTOMATE APACHE CERTIFICATES WITH CERTMONGER
#============================================#

# Install certmonger
# RHEL 8/9/10
sudo dnf install certmonger

# RHEL 7
# sudo yum install certmonger

sudo systemctl enable --now certmonger

# FreeIPA / internal CA workflow
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/www.example.com.crt \
  -k /etc/pki/tls/private/www.example.com.key \
  -D www.example.com \
  -K host/www.example.com@REALM \
  -C "systemctl reload httpd"  # Auto-reload Apache after renewal!

# Check status
sudo getcert list

# For public Let's Encrypt certificates, use certbot in section 14.7.

Benefits:

• ✅ Automatic renewal
• ✅ No downtime (reload, not restart)
• ✅ Tracks expiration
• ✅ Email alerts on failure

14.7 Let’s Encrypt with certbot

⚠️ IMPORTANT: EPEL Required

certbot is NOT available in official RHEL repositories. It requires EPEL (Extra Packages for Enterprise Linux), a community-maintained repository.

For production RHEL environments, consider:

• FreeIPA with certmonger (recommended for RHEL)
• Manual certificate management
• Commercial CA with certmonger

#============================================#
# CERTBOT SETUP (REQUIRES EPEL!)
#============================================#

# Step 1: Enable EPEL
# RHEL 7:
# sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm -y

# RHEL 8:
# sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm -y

# RHEL 9/10:
sudo dnf install epel-release -y

# Step 2: Install certbot
# RHEL 7:
# sudo yum install certbot python2-certbot-apache -y

# RHEL 8/9/10:
sudo dnf install certbot python3-certbot-apache -y

# Step 3: Obtain certificate (automated!)
sudo certbot --apache -d www.example.com -d example.com

# Step 4: Certbot automatically:
#  - Generates certificate
#  - Configures Apache
#  - Sets up renewal timer
#  - Enables HTTPS redirect

# Step 5: Test automatic renewal
sudo certbot renew --dry-run

# Check renewal timer
systemctl list-timers | grep certbot

Pros:

• ✅ Fully automated
• ✅ Free certificates
• ✅ Apache config handled automatically

Cons:

• ❌ Requires EPEL (not officially supported by Red Hat)
• ❌ External dependency (Let’s Encrypt)
• ⚠️ Domain must be publicly accessible

14.8 Troubleshooting Apache HTTPS

Common Issues Checklist

#============================================#
# APACHE SSL TROUBLESHOOTING CHECKLIST
#============================================#

# 1. Check if mod_ssl is loaded
sudo httpd -M | grep ssl_module
# Should show: ssl_module (shared)

# 2. Check configuration syntax
sudo apachectl configtest
# Should show: Syntax OK

# 3. Check certificate files exist
ls -l /etc/pki/tls/certs/www.crt
ls -l /etc/pki/tls/private/www.key

# 4. Verify permissions
ls -l /etc/pki/tls/private/www.key
# Should be: -rw------- (600)

# 5. Check SELinux context
ls -Z /etc/pki/tls/certs/www.crt
ls -Z /etc/pki/tls/private/www.key
# Should show: cert_t

# 6. Test certificate/key pair match
CERT_MOD=$(openssl x509 -noout -modulus -in /etc/pki/tls/certs/www.crt | openssl md5)
KEY_MOD=$(openssl rsa -noout -modulus -in /etc/pki/tls/private/www.key | openssl md5)
[ "$CERT_MOD" = "$KEY_MOD" ] && echo "✅ Match" || echo "❌ No match!"

# 7. Check if port 443 is listening
ss -tlnp | grep :443

# 8. Check firewall
sudo firewall-cmd --list-services | grep https

# 9. Test locally
curl -vk https://localhost/

# 10. Check logs
sudo tail -f /var/log/httpd/ssl_error_log

Common Errors and Solutions

14.9 Version-Specific Troubleshooting

RHEL 7 Specific

#============================================#
# RHEL 7 APACHE ISSUES
#============================================#

# Issue: Modern browsers reject TLS 1.0/1.1
# Solution: Disable old TLS in ssl.conf
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1

# Issue: Weak ciphers flagged by scan
# Solution: Use strong ciphers
SSLCipherSuite ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:HIGH:!aNULL:!MD5
SSLHonorCipherOrder on

# Issue: No SANs in certificate
# Solution: Reissue with SANs (see 13.5)

# Test
openssl s_client -connect localhost:443 -tls1_2

RHEL 8/9/10 Specific

#============================================#
# RHEL 8/9/10 APACHE ISSUES
#============================================#

# Issue: Service fails after crypto-policy change
# Diagnosis:
update-crypto-policies --show
sudo journalctl -xe -u httpd | grep -i tls

# Solution 1: Verify policy is correct
sudo update-crypto-policies --set DEFAULT

# Solution 2: Check if you manually override policy
grep -E "SSLProtocol|SSLCipherSuite" /etc/httpd/conf.d/*.conf
# If found, remove (let crypto-policy handle it)

# Issue: "no shared cipher" error
# Diagnosis: Client too old or policy too strict
# Temporary solution:
sudo update-crypto-policies --set LEGACY
sudo systemctl restart httpd

# Proper solution: Update client or create custom policy module

14.10 Security Best Practices

Hardened Apache SSL Configuration

#============================================#
# HARDENED SSL CONFIG (ALL VERSIONS)
#============================================#

<VirtualHost *:443>
    ServerName secure.example.com

    SSLEngine on
    SSLCertificateFile      /etc/pki/tls/certs/secure.crt
    SSLCertificateKeyFile   /etc/pki/tls/private/secure.key

    # RHEL 7: Manual TLS config
    # SSLProtocol TLSv1.2 TLSv1.3
    # SSLCipherSuite ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256
    # SSLHonorCipherOrder on

    # RHEL 8/9/10: Crypto-policies handle above automatically

    # HSTS (force HTTPS for 1 year)
    Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"

    # Prevent clickjacking
    Header always set X-Frame-Options "DENY"

    # Prevent MIME-type sniffing
    Header always set X-Content-Type-Options "nosniff"

    # Disable server signature
    ServerSignature Off
    ServerTokens Prod

    # OCSP Stapling (RHEL 8/9/10)
    SSLUseStapling on
    SSLStaplingCache "shmcb:/var/run/ocsp(128000)"

    # Client certificate auth (optional)
    # SSLVerifyClient require
    # SSLVerifyDepth 3
    # SSLCACertificateFile /etc/pki/tls/certs/client-ca.crt
</VirtualHost>

# Outside VirtualHost (global SSL settings)
SSLStaplingCache "shmcb:/var/run/ocsp(128000)"

14.11 HTTP to HTTPS Redirect

Force HTTPS

#============================================#
# REDIRECT HTTP → HTTPS
#============================================#

# Method 1: Separate VirtualHost
<VirtualHost *:80>
    ServerName www.example.com
    Redirect permanent / https://www.example.com/
</VirtualHost>

<VirtualHost *:443>
    ServerName www.example.com
    # ... SSL config ...
</VirtualHost>

# Method 2: mod_rewrite
<VirtualHost *:80>
    ServerName www.example.com

    RewriteEngine On
    RewriteCond %{HTTPS} off
    RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [R=301,L]
</VirtualHost>

14.12 Testing Apache HTTPS

Comprehensive Testing

#============================================#
# APACHE HTTPS TESTING SUITE
#============================================#

# Test 1: Configuration syntax
sudo apachectl configtest

# Test 2: SSL module loaded
sudo httpd -M | grep ssl

# Test 3: Port listening
ss -tlnp | grep :443

# Test 4: Local connection
curl -vk https://localhost/

# Test 5: Actual hostname
curl -v https://www.example.com/

# Test 6: Certificate validation
openssl s_client -connect www.example.com:443 -servername www.example.com

# Test 7: TLS 1.2
openssl s_client -connect www.example.com:443 -tls1_2

# Test 8: TLS 1.3 (RHEL 8+)
openssl s_client -connect www.example.com:443 -tls1_3

# Test 9: Check certificate details from server
echo | openssl s_client -connect www.example.com:443 -servername www.example.com 2>&1 | \
  openssl x509 -noout -text | head -30

# Test 10: Online test (external)
# Use: https://www.ssllabs.com/ssltest/

14.13 Performance Optimization

SSL/TLS Performance Tuning

#============================================#
# PERFORMANCE TUNING
#============================================#

<VirtualHost *:443>
    # ... basic config ...

    # Session cache (improves performance)
    SSLSessionCache         "shmcb:/var/cache/httpd/ssl_scache(512000)"
    SSLSessionCacheTimeout  300

    # OCSP Stapling (reduces client-side lookup)
    SSLUseStapling on
    SSLStaplingCache "shmcb:/var/run/ocsp(128000)"

    # Keep-Alive (reuse connections)
    KeepAlive On
    MaxKeepAliveRequests 100
    KeepAliveTimeout 5

    # HTTP/2 (RHEL 8/9/10)
    Protocols h2 h2c http/1.1
</VirtualHost>

14.14 Monitoring Apache HTTPS

What to Monitor

#============================================#
# APACHE HTTPS MONITORING
#============================================#

# Certificate expiration
openssl s_client -connect localhost:443 -servername $(hostname -f) 2>/dev/null | \
  openssl x509 -noout -dates

# Service status
systemctl status httpd

# Connection count
ss -tn | grep :443 | wc -l

# Error log monitoring
sudo tail -f /var/log/httpd/ssl_error_log

# certmonger status (if used)
sudo getcert list -f /etc/pki/tls/certs/www.crt

# Access log analysis
sudo tail -f /var/log/httpd/ssl_access_log | grep -E "HTTP/[12]"

14.15 Quick Troubleshooting Guide

Apache HTTPS Not Working?

├─ Apache won't start?
│  ├─ Check: apachectl configtest
│  ├─ Check: journalctl -xe -u httpd
│  └─ Fix: Configuration errors
│
├─ Can't connect to port 443?
│  ├─ Check: ss -tlnp | grep :443
│  ├─ Check: firewall-cmd --list-services
│  └─ Fix: Open firewall, start httpd
│
├─ Certificate warnings in browser?
│  ├─ Check: Certificate expiration
│  ├─ Check: Hostname match (CN/SANs)
│  ├─ Check: Trust chain
│  └─ Fix: Renew cert, fix SANs, install CA
│
├─ "No shared cipher" error?
│  ├─ Check: update-crypto-policies --show
│  ├─ Check: Client TLS version
│  └─ Fix: Update policy or client
│
└─ Permission errors?
   ├─ Check: ls -lZ /etc/pki/tls/private/*.key
   ├─ Check: SELinux denials
   └─ Fix: chmod 600, restorecon

14.16 Key Takeaways

1. Apache + mod_ssl is the standard RHEL web server
2. RHEL 7: Manual TLS configuration required
3. RHEL 8/9/10: Crypto-policies simplify configuration
4. certmonger integration enables automation
5. certbot requires EPEL (not officially supported)
6. Always use SANs in certificates
7. Test thoroughly before production deployment

Quick Reference Card

┌───────────────────────────────────────────────────────────────────┐
│ APACHE HTTPD HTTPS QUICK REFERENCE                                │
├───────────────────────────────────────────────────────────────────┤
│ Install:      dnf install httpd mod_ssl                           │
│ Config:       /etc/httpd/conf.d/ssl.conf                          │
│ Certs:        /etc/pki/tls/certs/                                 │
│ Keys:         /etc/pki/tls/private/ (mode 600!)                   │
│                                                                   │
│ Test config:  apachectl configtest                                │
│ Reload:       systemctl reload httpd                              │
│ Logs:         /var/log/httpd/ssl_error_log                        │
│                                                                   │
│ certmonger:   ipa-getcert request ... -C "systemctl reload httpd" │
│ certbot:      certbot --apache (requires EPEL!)                   │
│                                                                   │
│ Test:         curl -v https://localhost/                          │
│               openssl s_client -connect host:443                  │
└───────────────────────────────────────────────────────────────────┘

⚠️ RHEL 8/9/10: Let crypto-policies handle TLS/cipher config
⚠️ certbot requires EPEL (not officially supported)

🧪 Hands-On Lab

Lab 06: Apache HTTPS Setup

Configure Apache with SSL/TLS across RHEL versions

• 📁 Location: labs/en_US/06-apache-https/
• ⏱️ Time: 30-40 minutes
• 🎯 Level: Intermediate

Chapter Navigation

Chapter 15: NGINX on RHEL

High Performance: NGINX is a popular high-performance web server and reverse proxy. Learn how to configure NGINX with TLS certificates on RHEL.

15.1 NGINX on RHEL Overview

Package Name: nginx Config Location: /etc/nginx/nginx.conf Certificate Path: /etc/pki/tls/certs/ or /etc/nginx/certs/ Key Path: /etc/pki/tls/private/ or /etc/nginx/certs/

Installation Sources by RHEL Version

Note: RHEL 7 requires EPEL for NGINX. RHEL 8+ includes NGINX in official repos.

15.2 Installation

RHEL 7

#============================================#
# INSTALL NGINX (RHEL 7 - REQUIRES EPEL)
#============================================#

# Step 1: Enable EPEL
sudo yum install epel-release -y

# Step 2: Install NGINX
sudo yum install nginx -y

# Step 3: Enable and start
sudo systemctl enable nginx
sudo systemctl start nginx

# Step 4: Open firewall
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

# Verify
systemctl status nginx
curl http://localhost/

RHEL 8

#============================================#
# INSTALL NGINX (RHEL 8 - FROM APPSTREAM)
#============================================#

# List available NGINX modules
dnf module list nginx

# Install specific version
sudo dnf module install nginx:1.20 -y

# Or install default
sudo dnf install nginx -y

# Enable and start
sudo systemctl enable nginx
sudo systemctl start nginx

# Open firewall
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

RHEL 9/10

#============================================#
# INSTALL NGINX (RHEL 9/10)
#============================================#

sudo dnf install nginx -y

sudo systemctl enable nginx
sudo systemctl start nginx

# Open firewall
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

# Verify
systemctl status nginx
ss -tlnp | grep :443

15.3 Basic HTTPS Configuration

Minimal TLS Setup

#============================================#
# /etc/nginx/nginx.conf or /etc/nginx/conf.d/default.conf
#============================================#

server {
    listen 80;
    server_name www.example.com;

    # Redirect HTTP to HTTPS
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name www.example.com;

    # Certificate files
    ssl_certificate     /etc/pki/tls/certs/www.example.com.crt;
    ssl_certificate_key /etc/pki/tls/private/www.example.com.key;

    # TLS protocols
    ssl_protocols TLSv1.2 TLSv1.3;

    # Ciphers
    ssl_ciphers HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers on;

    # Root directory
    root /usr/share/nginx/html;
    index index.html;

    location / {
        try_files $uri $uri/ =404;
    }
}

Hardened Production Configuration

#============================================#
# PRODUCTION-GRADE NGINX HTTPS CONFIG
#============================================#

server {
    listen 443 ssl http2;
    server_name api.example.com;

    # Certificates
    ssl_certificate     /etc/pki/tls/certs/api.example.com.crt;
    ssl_certificate_key /etc/pki/tls/private/api.example.com.key;

    # TLS versions
    ssl_protocols TLSv1.2 TLSv1.3;

    # Strong ciphers
    ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256';
    ssl_prefer_server_ciphers on;

    # DH parameters (optional, for perfect forward secrecy)
    ssl_dhparam /etc/nginx/dhparam.pem;

    # SSL session optimization
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    ssl_session_tickets off;

    # OCSP Stapling
    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /etc/pki/tls/certs/chain.crt;
    resolver 8.8.8.8 8.8.4.4 valid=300s;
    resolver_timeout 5s;

    # HSTS
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;

    # Security headers
    add_header X-Frame-Options DENY always;
    add_header X-Content-Type-Options nosniff always;
    add_header X-XSS-Protection "1; mode=block" always;

    # Logging
    access_log /var/log/nginx/api_access.log;
    error_log /var/log/nginx/api_error.log;

    location / {
        proxy_pass http://backend_servers;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

15.4 Certificate Setup

Generate Certificates for NGINX

#============================================#
# CERTIFICATE GENERATION FOR NGINX
#============================================#

# Step 1: Create certs directory (optional)
sudo mkdir -p /etc/nginx/certs
sudo chmod 755 /etc/nginx/certs

# Step 2: Generate private key
sudo openssl genpkey -algorithm RSA \
  -out /etc/pki/tls/private/api.example.com.key \
  -pkeyopt rsa_keygen_bits:2048

# Step 3: Set permissions
sudo chmod 600 /etc/pki/tls/private/api.example.com.key
sudo chown root:nginx /etc/pki/tls/private/api.example.com.key

# Step 4: Generate CSR
sudo openssl req -new \
  -key /etc/pki/tls/private/api.example.com.key \
  -out /tmp/api.example.com.csr \
  -subj "/CN=api.example.com" \
  -addext "subjectAltName=DNS:api.example.com,DNS:www.api.example.com"

# Step 5: Submit to CA, receive certificate

# Step 6: Install certificate
sudo cp api.example.com.crt /etc/pki/tls/certs/
sudo chmod 644 /etc/pki/tls/certs/api.example.com.crt

15.5 certmonger Integration

Automated Renewal with certmonger

#============================================#
# CERTMONGER + NGINX
#============================================#

# Install certmonger
# RHEL 8/9/10
sudo dnf install certmonger

# RHEL 7
# sudo yum install certmonger

sudo systemctl enable --now certmonger

# Request certificate from FreeIPA / internal CA
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/nginx.example.com.crt \
  -k /etc/pki/tls/private/nginx.example.com.key \
  -D nginx.example.com \
  -K host/nginx.example.com@REALM \
  -C "systemctl reload nginx"  # Auto-reload on renewal!

# Monitor status
sudo getcert list

# For public Let's Encrypt certificates, use certbot in section 15.6.

15.6 Let’s Encrypt with certbot

⚠️ IMPORTANT: EPEL Required

certbot is NOT available in official RHEL repositories. It requires EPEL, a community-maintained repository.

Installation: All RHEL versions require EPEL, but the enablement command differs by version. See Chapter 24 for the full certbot workflow.

RHEL 7

# Step 1: Enable EPEL
sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm -y

# Step 2: Install certbot with NGINX plugin
sudo yum install certbot python2-certbot-nginx -y

RHEL 8

# Step 1: Enable EPEL
sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm -y
# Or, with an active subscription:
# sudo dnf install epel-release -y

# Step 2: Install certbot with NGINX plugin
sudo dnf install certbot python3-certbot-nginx -y

RHEL 9/10

# Step 1: Enable EPEL
sudo dnf install epel-release -y

# Step 2: Install certbot with NGINX plugin
sudo dnf install certbot python3-certbot-nginx -y

Obtain and configure the certificate (all versions)

# Step 3: Obtain and install certificate (automated!)
sudo certbot --nginx -d www.example.com -d example.com

# Certbot will:
#  ✅ Generate certificate from Let's Encrypt
#  ✅ Update NGINX configuration automatically
#  ✅ Set up HTTP to HTTPS redirect
#  ✅ Configure auto-renewal

# Step 4: Verify auto-renewal timer
systemctl list-timers | grep certbot

# Step 5: Test renewal (dry run)
sudo certbot renew --dry-run

# Certificate renews automatically every 60 days!

Remember: EPEL is community-supported, not Red Hat supported. For enterprise production, consider FreeIPA + certmonger.

15.7 Reverse Proxy with TLS

NGINX as TLS Termination Proxy

#============================================#
# NGINX REVERSE PROXY WITH TLS
#============================================#

upstream backend_servers {
    server 10.0.1.10:8080;
    server 10.0.1.11:8080;
    server 10.0.1.12:8080;
}

server {
    listen 443 ssl http2;
    server_name proxy.example.com;

    # TLS termination here
    ssl_certificate     /etc/pki/tls/certs/proxy.crt;
    ssl_certificate_key /etc/pki/tls/private/proxy.key;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;

    # Proxy to backends (HTTP)
    location / {
        proxy_pass http://backend_servers;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
    }
}

15.8 Troubleshooting NGINX HTTPS

Diagnostic Commands

#============================================#
# NGINX HTTPS TROUBLESHOOTING
#============================================#

# Test configuration syntax
sudo nginx -t

# Show full configuration (with includes)
sudo nginx -T

# Check SSL certificate paths
sudo nginx -T | grep ssl_certificate

# Verify certificate file
sudo openssl x509 -in /etc/pki/tls/certs/nginx.crt -noout -text

# Verify key file
sudo openssl rsa -in /etc/pki/tls/private/nginx.key -check

# Check cert/key pair match
CERT=$(openssl x509 -noout -modulus -in /etc/pki/tls/certs/nginx.crt | openssl md5)
KEY=$(openssl rsa -noout -modulus -in /etc/pki/tls/private/nginx.key | openssl md5)
[ "$CERT" = "$KEY" ] && echo "✅ Match" || echo "❌ Mismatch!"

# Check if NGINX is listening on 443
ss -tlnp | grep :443

# Check SELinux context
ls -Z /etc/pki/tls/certs/nginx.crt
ls -Z /etc/pki/tls/private/nginx.key

# Test HTTPS locally
curl -vk https://localhost/

# Check logs
sudo tail -f /var/log/nginx/error.log

Common NGINX HTTPS Errors

15.9 Version-Specific Configuration

RHEL 7: Manual TLS Configuration

#============================================#
# NGINX RHEL 7 - MANUAL SSL CONFIG
#============================================#

server {
    listen 443 ssl;
    server_name www.example.com;

    ssl_certificate     /etc/pki/tls/certs/www.crt;
    ssl_certificate_key /etc/pki/tls/private/www.key;

    # REQUIRED: Manually disable weak TLS
    ssl_protocols TLSv1.2;  # No TLS 1.0/1.1!

    # REQUIRED: Manually set strong ciphers
    ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:HIGH:!aNULL:!MD5';
    ssl_prefer_server_ciphers on;

    # Security headers
    add_header Strict-Transport-Security "max-age=31536000" always;

    root /usr/share/nginx/html;
}

RHEL 8/9/10: Crypto-Policies Aware

#============================================#
# NGINX RHEL 8/9/10 - WITH CRYPTO-POLICIES
#============================================#

server {
    listen 443 ssl http2;
    server_name www.example.com;

    ssl_certificate     /etc/pki/tls/certs/www.crt;
    ssl_certificate_key /etc/pki/tls/private/www.key;

    # Minimal TLS config - crypto-policies handle the rest!
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers on;

    # Or completely rely on crypto-policies:
    # (remove ssl_protocols and ssl_ciphers)
    # NGINX will use system crypto-policy

    # Session optimization
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;

    # OCSP Stapling
    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /etc/pki/tls/certs/chain.crt;

    # HSTS
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

    root /usr/share/nginx/html;
}

15.10 Performance Optimization

SSL/TLS Performance Tuning

#============================================#
# NGINX SSL PERFORMANCE OPTIMIZATION
#============================================#

http {
    # SSL session cache (reduces handshake overhead)
    ssl_session_cache shared:SSL:50m;
    ssl_session_timeout 1d;
    ssl_session_tickets off;

    # Buffer sizes
    ssl_buffer_size 4k;  # Smaller = lower latency, larger = better throughput

    server {
        listen 443 ssl http2;
        server_name fast.example.com;

        ssl_certificate     /etc/pki/tls/certs/fast.crt;
        ssl_certificate_key /etc/pki/tls/private/fast.key;

        # Use HTTP/2 for multiplexing
        # (already enabled in listen directive)

        # Enable OCSP Stapling (reduces client lookup time)
        ssl_stapling on;
        ssl_stapling_verify on;

        # Keep-alive
        keepalive_timeout 70;
        keepalive_requests 100;

        location / {
            proxy_pass http://backend;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
        }
    }
}

15.11 Multiple Certificates (SNI)

Server Name Indication (SNI)

#============================================#
# MULTIPLE DOMAINS WITH DIFFERENT CERTIFICATES
#============================================#

# Site 1
server {
    listen 443 ssl http2;
    server_name site1.example.com;

    ssl_certificate     /etc/pki/tls/certs/site1.crt;
    ssl_certificate_key /etc/pki/tls/private/site1.key;

    root /var/www/site1;
}

# Site 2
server {
    listen 443 ssl http2;
    server_name site2.example.com;

    ssl_certificate     /etc/pki/tls/certs/site2.crt;
    ssl_certificate_key /etc/pki/tls/private/site2.key;

    root /var/www/site2;
}

# Site 3 (wildcard)
server {
    listen 443 ssl http2;
    server_name *.apps.example.com;

    ssl_certificate     /etc/pki/tls/certs/wildcard.apps.crt;
    ssl_certificate_key /etc/pki/tls/private/wildcard.apps.key;

    root /var/www/apps;
}

15.12 Client Certificate Authentication

Mutual TLS (mTLS) with NGINX

#============================================#
# NGINX WITH CLIENT CERTIFICATE AUTH
#============================================#

server {
    listen 443 ssl http2;
    server_name secure.example.com;

    # Server certificates
    ssl_certificate     /etc/pki/tls/certs/secure.crt;
    ssl_certificate_key /etc/pki/tls/private/secure.key;

    # Client certificate verification
    ssl_client_certificate /etc/pki/tls/certs/client-ca.crt;
    ssl_verify_client on;  # or 'optional'
    ssl_verify_depth 3;

    # Pass client cert info to backend
    location / {
        proxy_pass http://backend;
        proxy_set_header X-SSL-Client-Cert $ssl_client_cert;
        proxy_set_header X-SSL-Client-DN $ssl_client_s_dn;
        proxy_set_header X-SSL-Client-Verify $ssl_client_verify;
    }
}

15.13 Testing NGINX HTTPS

Comprehensive Test Suite

#============================================#
# NGINX HTTPS TESTING
#============================================#

# Test 1: Configuration syntax
sudo nginx -t
# nginx: configuration file /etc/nginx/nginx.conf test is successful

# Test 2: Show effective configuration
sudo nginx -T | grep -A10 "server_name www.example.com"

# Test 3: Port listening
ss -tlnp | grep nginx

# Test 4: Local HTTPS
curl -vk https://localhost/

# Test 5: Hostname-based
curl -v https://www.example.com/

# Test 6: Check certificate from server
echo | openssl s_client -connect www.example.com:443 -servername www.example.com 2>&1 | \
  openssl x509 -noout -subject -dates

# Test 7: TLS 1.2
openssl s_client -connect www.example.com:443 -tls1_2

# Test 8: TLS 1.3 (RHEL 8+)
openssl s_client -connect www.example.com:443 -tls1_3

# Test 9: HTTP/2 support
curl -I --http2 https://www.example.com/

# Test 10: Security headers
curl -I https://www.example.com/ | grep -i "strict-transport"

15.14 Common Issues and Solutions

Issue 1: “Permission denied” on Private Key

# Symptom
sudo nginx -t
# nginx: [emerg] SSL_CTX_use_PrivateKey_file() failed (SSL: error:0200100D:system library:fopen:Permission denied)

# Check permissions
ls -l /etc/pki/tls/private/nginx.key

# Fix
sudo chmod 600 /etc/pki/tls/private/nginx.key
sudo chown root:nginx /etc/pki/tls/private/nginx.key

# If SELinux issue:
sudo restorecon -v /etc/pki/tls/private/nginx.key

Issue 2: Certificate Chain Not Sent

# Test from client
openssl s_client -connect www.example.com:443 -showcerts

# If shows only server cert (not intermediates):
# Create certificate bundle
cat server.crt intermediate.crt > /etc/pki/tls/certs/bundle.crt

# Update NGINX config
ssl_certificate /etc/pki/tls/certs/bundle.crt;

# Reload
sudo systemctl reload nginx

Issue 3: OCSP Stapling Not Working

# Test OCSP stapling
openssl s_client -connect www.example.com:443 -status -tlsextdebug 2>&1 | grep -A17 "OCSP"

# Common causes:
# 1. No resolver configured
# Fix: Add to nginx.conf:
resolver 8.8.8.8 8.8.4.4 valid=300s;

# 2. Missing trusted certificate chain
# Fix:
ssl_trusted_certificate /etc/pki/tls/certs/chain.crt;

# 3. Firewall blocks OCSP requests
# Fix: Allow outbound HTTPS

15.15 Security Best Practices

Checklist

✅ Use TLS 1.2+ only (disable 1.0/1.1)
✅ Strong ciphers with forward secrecy (ECDHE)
✅ Enable HSTS with long max-age
✅ Enable OCSP Stapling
✅ Use HTTP/2
✅ Proper file permissions (600 for keys)
✅ SELinux contexts correct
✅ Certificate validity ≤ 90 days with auto-renewal
✅ Include comprehensive SANs
✅ Security headers enabled

15.16 Key Takeaways

1. NGINX available in AppStream (RHEL 8+) or EPEL (RHEL 7)
2. Crypto-policies simplify config on RHEL 8/9/10
3. certmonger integrates well with automatic reload
4. certbot requires EPEL on all RHEL versions
5. SNI enables multiple certs on same IP
6. mTLS possible for client authentication
7. Test thoroughly - syntax, connectivity, security

Quick Reference Card

┌──────────────────────────────────────────────────────────────┐
│ NGINX HTTPS QUICK REFERENCE                                  │
├──────────────────────────────────────────────────────────────┤
│ Install:      dnf install nginx (RHEL 8/9/10)                │
│               yum install epel-release nginx (RHEL 7)        │
│                                                              │
│ Config:       /etc/nginx/nginx.conf                          │
│               /etc/nginx/conf.d/*.conf                       │
│                                                              │
│ Basic SSL:    listen 443 ssl http2;                          │
│               ssl_certificate /path/to/cert.crt;             │
│               ssl_certificate_key /path/to/key.key;          │
│                                                              │
│ Test:         nginx -t                                       │
│ Reload:       systemctl reload nginx                         │
│ Logs:         /var/log/nginx/error.log                       │
│                                                              │
│ certbot:      certbot --nginx (requires EPEL!)               │
│ certmonger:   ipa-getcert ... -C "systemctl reload nginx"    │
└──────────────────────────────────────────────────────────────┘

⚠️ certbot requires EPEL on all RHEL versions
✅ Use certmonger for enterprise environments

🧪 Hands-On Lab

Lab 07: NGINX HTTPS Setup

Configure NGINX with SSL/TLS and security best practices

• 📁 Location: labs/en_US/07-nginx-https/
• ⏱️ Time: 30-35 minutes
• 🎯 Level: Intermediate

Chapter Navigation

Chapter 16: Postfix Mail Server TLS

Email Security: Learn how to configure TLS encryption for Postfix mail servers on RHEL, protecting email communications with certificates.

16.1 Postfix on RHEL Overview

Package Name: postfix Config Location: /etc/postfix/main.cf Certificate Path: /etc/pki/tls/certs/ Key Path: /etc/pki/tls/private/ Ports: 25 (SMTP), 465 (SMTPS), 587 (Submission with STARTTLS)

Why TLS for Email?

• ✅ Encrypt email in transit (prevent eavesdropping)
• ✅ Authenticate mail servers (prevent impersonation)
• ✅ Meet compliance requirements (HIPAA, PCI-DSS)
• ✅ Prevent spam/phishing (SPF, DKIM, DMARC work better with TLS)

16.2 Installation

All RHEL Versions

#============================================#
# INSTALL POSTFIX
#============================================#

# Install Postfix
sudo dnf install postfix -y

# Stop and disable Sendmail (if present)
sudo systemctl stop sendmail 2>/dev/null
sudo systemctl disable sendmail 2>/dev/null

# Enable Postfix
sudo systemctl enable postfix
sudo systemctl start postfix

# Open firewall
sudo firewall-cmd --permanent --add-service=smtp
sudo firewall-cmd --permanent --add-service=smtps
sudo firewall-cmd --permanent --add-service=smtp-submission
sudo firewall-cmd --reload

# Verify
systemctl status postfix
ss -tlnp | grep -E ':(25|465|587)'

16.3 Server-Side TLS Configuration

Receiving Mail with TLS (SMTPD)

#============================================#
# /etc/postfix/main.cf - SERVER (RECEIVING) TLS
#============================================#

# Certificate files
smtpd_tls_cert_file = /etc/pki/tls/certs/mail.example.com.crt
smtpd_tls_key_file = /etc/pki/tls/private/mail.example.com.key
smtpd_tls_CAfile = /etc/pki/tls/certs/ca-bundle.crt

# TLS security level
smtpd_tls_security_level = may  # or 'encrypt' to require TLS

# Logging
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes

# Session cache (performance)
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache

# Authentication - only over TLS
smtpd_tls_auth_only = yes

# RHEL 7: Specify protocols manually
# smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
# smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1

# RHEL 8/9/10: Crypto-policies handle protocols automatically
# (no need to specify smtpd_tls_protocols)

Security Levels Explained:

16.4 Client-Side TLS Configuration

Sending Mail with TLS (SMTP)

#============================================#
# /etc/postfix/main.cf - CLIENT (SENDING) TLS
#============================================#

# Certificate files (optional for client)
smtp_tls_cert_file = /etc/pki/tls/certs/mail.example.com.crt
smtp_tls_key_file = /etc/pki/tls/private/mail.example.com.key
smtp_tls_CAfile = /etc/pki/tls/certs/ca-bundle.crt

# TLS security level for outbound
smtp_tls_security_level = may  # or 'encrypt' to require

# Logging
smtp_tls_loglevel = 1

# Session cache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

# RHEL 7: Specify protocols
# smtp_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
# smtp_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1

# RHEL 8/9/10: Crypto-policies handle it

16.5 Complete Postfix TLS Setup

Full Configuration Example

#============================================#
# COMPLETE POSTFIX TLS CONFIGURATION
#============================================#

# 1. Generate certificate and key
sudo openssl genpkey -algorithm RSA \
  -out /etc/pki/tls/private/mail.example.com.key \
  -pkeyopt rsa_keygen_bits:2048

sudo chmod 600 /etc/pki/tls/private/mail.example.com.key

# 2. Generate CSR
sudo openssl req -new \
  -key /etc/pki/tls/private/mail.example.com.key \
  -out /tmp/mail.example.com.csr \
  -subj "/CN=mail.example.com" \
  -addext "subjectAltName=DNS:mail.example.com,DNS:smtp.example.com"

# 3. Get certificate from CA, install it
sudo cp mail.example.com.crt /etc/pki/tls/certs/
sudo chmod 644 /etc/pki/tls/certs/mail.example.com.crt

# 4. Edit /etc/postfix/main.cf
sudo vi /etc/postfix/main.cf

# Add TLS configuration (see sections 16.3 and 16.4)

# 5. Test configuration
sudo postfix check

# 6. Reload Postfix
sudo systemctl reload postfix

# 7. Test SMTP TLS
openssl s_client -starttls smtp -connect mail.example.com:25

16.6 SMTPS (Port 465) Configuration

Enable SMTPS Service

#============================================#
# /etc/postfix/master.cf - ENABLE SMTPS
#============================================#

# Uncomment or add SMTPS service (port 465)
smtps     inet  n       -       n       -       -       smtpd
  -o syslog_name=postfix/smtps
  -o smtpd_tls_wrappermode=yes
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING

# Reload
sudo systemctl reload postfix

# Verify
ss -tlnp | grep :465

Test SMTPS

# Connect to SMTPS (immediate TLS, no STARTTLS)
openssl s_client -connect mail.example.com:465

# Should show TLS handshake immediately

16.7 Submission Port (587) with STARTTLS

Enable Submission Service

#============================================#
# /etc/postfix/master.cf - ENABLE SUBMISSION
#============================================#

# Uncomment or add submission service (port 587)
submission inet n       -       n       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING

# Reload
sudo systemctl reload postfix

# Verify
ss -tlnp | grep :587

Test Submission Port

# Test STARTTLS on port 587
openssl s_client -starttls smtp -connect mail.example.com:587

# Should show:
# STARTTLS
# 220 2.0.0 Ready to start TLS

16.8 certmonger Integration

Automated Certificate Management for Postfix

#============================================#
# CERTMONGER + POSTFIX
#============================================#

# Install certmonger
# RHEL 8/9/10
sudo dnf install certmonger

# RHEL 7
# sudo yum install certmonger
sudo systemctl enable --now certmonger

# Request certificate from FreeIPA
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/mail.example.com.crt \
  -k /etc/pki/tls/private/mail.example.com.key \
  -D mail.example.com \
  -K host/mail.example.com@REALM \
  -C "postfix reload"  # Auto-reload Postfix after renewal

# Check status
sudo getcert list

# Postfix automatically uses renewed certificate!

16.9 Client Certificate Authentication

Require Client Certificates (mTLS for SMTP)

#============================================#
# /etc/postfix/main.cf - CLIENT CERT AUTH
#============================================#

# Server certificates (as before)
smtpd_tls_cert_file = /etc/pki/tls/certs/mail.crt
smtpd_tls_key_file = /etc/pki/tls/private/mail.key

# Client certificate verification
smtpd_tls_CAfile = /etc/pki/tls/certs/client-ca.crt
smtpd_tls_ask_ccert = yes
smtpd_tls_req_ccert = yes  # Require client cert

# Map certificate to user
smtpd_recipient_restrictions =
  permit_tls_clientcerts,
  reject

# Reload
sudo postfix reload

Test with client certificate:

openssl s_client -starttls smtp \
  -connect mail.example.com:25 \
  -cert client.crt \
  -key client.key \
  -CAfile ca.crt

16.10 Troubleshooting Postfix TLS

Diagnostic Commands

#============================================#
# POSTFIX TLS TROUBLESHOOTING
#============================================#

# Check Postfix configuration
sudo postconf | grep -i tls

# Test configuration syntax
sudo postfix check

# View specific TLS settings
sudo postconf smtpd_tls_cert_file smtpd_tls_key_file

# Check certificate files exist
ls -l /etc/pki/tls/certs/mail.crt
ls -l /etc/pki/tls/private/mail.key

# Verify cert/key pair match
CERT_MOD=$(openssl x509 -noout -modulus -in /etc/pki/tls/certs/mail.crt | openssl md5)
KEY_MOD=$(openssl rsa -noout -modulus -in /etc/pki/tls/private/mail.key | openssl md5)
[ "$CERT_MOD" = "$KEY_MOD" ] && echo "✅ Match" || echo "❌ Mismatch!"

# Test SMTP STARTTLS
openssl s_client -starttls smtp -connect mail.example.com:25

# Test SMTPS (port 465)
openssl s_client -connect mail.example.com:465

# Check logs
sudo tail -f /var/log/maillog | grep -i tls

# Check Postfix queue
mailq

# Verbose logging (for troubleshooting)
sudo postconf smtpd_tls_loglevel=2
sudo postfix reload
# Check /var/log/maillog for detailed TLS info

Common Postfix TLS Issues

16.11 Version-Specific Considerations

RHEL 7

#============================================#
# POSTFIX TLS - RHEL 7 SPECIFIC
#============================================#

# /etc/postfix/main.cf

# MUST manually specify TLS protocols
smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1

smtp_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtp_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1

# MUST manually specify ciphers
smtpd_tls_mandatory_ciphers = high
smtpd_tls_ciphers = high

smtp_tls_mandatory_ciphers = high
smtp_tls_ciphers = high

# Exclude weak ciphers
smtpd_tls_mandatory_exclude_ciphers = aNULL, eNULL, EXPORT, DES, RC4, MD5, PSK, aECDH, EDH-DSS-DES-CBC3-SHA, EDH-RSA-DES-CBC3-SHA, KRB5-DES, CBC3-SHA

RHEL 8/9/10

#============================================#
# POSTFIX TLS - RHEL 8/9/10 WITH CRYPTO-POLICIES
#============================================#

# /etc/postfix/main.cf

# Certificate files
smtpd_tls_cert_file = /etc/pki/tls/certs/mail.crt
smtpd_tls_key_file = /etc/pki/tls/private/mail.key
smtpd_tls_CAfile = /etc/pki/tls/certs/ca-bundle.crt

# TLS settings (crypto-policies handle protocols/ciphers!)
smtpd_tls_security_level = may
smtpd_tls_auth_only = yes
smtpd_tls_loglevel = 1

# Client TLS
smtp_tls_cert_file = /etc/pki/tls/certs/mail.crt
smtp_tls_key_file = /etc/pki/tls/private/mail.key
smtp_tls_security_level = may
smtp_tls_loglevel = 1

# That's it! Much simpler than RHEL 7
# crypto-policies automatically configure TLS versions and ciphers

Key Difference: RHEL 8+ relies on crypto-policies, making configuration much simpler!

16.12 Testing Postfix TLS

Comprehensive Test Suite

#============================================#
# POSTFIX TLS TESTING
#============================================#

# Test 1: STARTTLS on port 25
openssl s_client -starttls smtp -connect mail.example.com:25

# Look for:
# - "250-STARTTLS" in server capabilities
# - Successful TLS handshake
# - Certificate details

# Test 2: SMTPS on port 465 (direct TLS)
openssl s_client -connect mail.example.com:465

# Test 3: Submission port 587 with STARTTLS
openssl s_client -starttls smtp -connect mail.example.com:587

# Test 4: Verify certificate from server
echo "QUIT" | openssl s_client -starttls smtp -connect mail.example.com:25 2>&1 | \
  openssl x509 -noout -subject -dates

# Test 5: Send test email
echo "Test email" | mail -s "TLS Test" test@example.com

# Test 6: Check TLS in logs
sudo tail -f /var/log/maillog | grep TLS

# Test 7: Check if TLS is being used
sudo postconf | grep -i tls | grep -i level

16.13 Security Best Practices

Hardened Postfix TLS Configuration

#============================================#
# HARDENED POSTFIX TLS CONFIG
#============================================#

# /etc/postfix/main.cf

# Server TLS (receiving mail)
smtpd_tls_cert_file = /etc/pki/tls/certs/mail.crt
smtpd_tls_key_file = /etc/pki/tls/private/mail.key
smtpd_tls_CAfile = /etc/pki/tls/certs/ca-bundle.crt

# REQUIRE TLS for authentication
smtpd_tls_security_level = may
smtpd_tls_auth_only = yes

# Client TLS (sending mail)
smtp_tls_cert_file = /etc/pki/tls/certs/mail.crt
smtp_tls_key_file = /etc/pki/tls/private/mail.key
smtp_tls_security_level = may

# RHEL 7: Manually disable weak protocols
smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtp_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1

# Mandatory for authenticated connections
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1

# Cipher settings (RHEL 7)
smtpd_tls_mandatory_ciphers = high
smtpd_tls_exclude_ciphers = aNULL, eNULL, EXPORT, DES, RC4, MD5, PSK, aECDH, EDH-DSS-DES-CBC3-SHA, EDH-RSA-DES-CBC3-SHA, KRB5-DES, CBC3-SHA

# Session caching
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

# Enhanced logging (temporary for troubleshooting)
smtpd_tls_loglevel = 1
smtp_tls_loglevel = 1

# Performance
smtpd_tls_session_cache_timeout = 3600s
tls_random_source = dev:/dev/urandom

16.14 Dovecot IMAP/POP3 with TLS

Dovecot Configuration

While this chapter focuses on Postfix (SMTP), Dovecot often runs alongside for IMAP/POP3:

#============================================#
# DOVECOT TLS (/etc/dovecot/conf.d/10-ssl.conf)
#============================================#

# Enable SSL
ssl = required

# Certificate files
ssl_cert = </etc/pki/tls/certs/mail.example.com.crt
ssl_key = </etc/pki/tls/private/mail.example.com.key

# CA for client cert verification (optional)
#ssl_ca = </etc/pki/tls/certs/ca-bundle.crt

# RHEL 8/9/10: crypto-policies handle it

# TLS protocols (RHEL 7)
#ssl_min_protocol = TLSv1.2

# Cipher suite preferences
#ssl_cipher_list = HIGH:!aNULL:!MD5

# Reload
sudo systemctl reload dovecot

Test IMAPS:

openssl s_client -connect mail.example.com:993

# Test POP3S
openssl s_client -connect mail.example.com:995

16.15 Common Scenarios

Scenario 1: Opportunistic TLS (Standard)

Goal: Accept both encrypted and unencrypted connections

# /etc/postfix/main.cf
smtpd_tls_security_level = may  # Optional TLS
smtp_tls_security_level = may   # Optional TLS

# Why: Maximum compatibility
# Drawback: Some connections may not be encrypted

Scenario 2: Mandatory TLS (High Security)

Goal: Reject unencrypted connections

# /etc/postfix/main.cf
smtpd_tls_security_level = encrypt  # REQUIRE TLS
smtp_tls_security_level = encrypt   # REQUIRE TLS

# Why: Maximum security
# Drawback: Old systems may fail to connect

Scenario 3: Client Certificate for Relay

Goal: Allow relay only with valid client certificate

# /etc/postfix/main.cf
smtpd_tls_cert_file = /etc/pki/tls/certs/mail.crt
smtpd_tls_key_file = /etc/pki/tls/private/mail.key
smtpd_tls_CAfile = /etc/pki/tls/certs/client-ca.crt

smtpd_tls_ask_ccert = yes
smtpd_tls_req_ccert = yes

smtpd_recipient_restrictions =
  permit_tls_clientcerts,
  reject_unauth_destination

# Only clients with valid cert from client-ca.crt can relay

16.16 Monitoring Postfix TLS

What to Monitor

#============================================#
# POSTFIX TLS MONITORING
#============================================#

# Check TLS usage in logs
sudo grep "TLS connection established" /var/log/maillog | tail -20

# Count TLS vs non-TLS connections
sudo grep "connect from" /var/log/maillog | grep -c "TLS"

# Check for TLS errors
sudo grep "TLS" /var/log/maillog | grep -i error

# Monitor certificate expiration
openssl x509 -in /etc/pki/tls/certs/mail.crt -noout -checkend $((86400*30))

# certmonger status (if used)
sudo getcert list -f /etc/pki/tls/certs/mail.crt

# Check queue for TLS failures
mailq

Monitoring Script

#!/bin/bash
# monitor-postfix-tls.sh

echo "=== Postfix TLS Monitoring ==="

# Certificate expiration
echo "1. Certificate Expiration:"
openssl x509 -in /etc/pki/tls/certs/mail.crt -noout -enddate

# TLS connections today
echo ""
echo "2. TLS Connections Today:"
sudo grep "$(date '+%b %e')" /var/log/maillog | grep "TLS connection" | wc -l

# TLS errors today
echo ""
echo "3. TLS Errors Today:"
sudo grep "$(date '+%b %e')" /var/log/maillog | grep -i "tls.*error" | tail -5

# Service status
echo ""
echo "4. Postfix Status:"
systemctl is-active postfix

# Listening ports
echo ""
echo "5. Listening on:"
ss -tlnp | grep master | grep -E ':(25|465|587)'

16.17 Common Issues and Solutions

Issue 1: Certificate Not Trusted by Clients

Symptom: Clients get “untrusted certificate” warnings

Diagnosis:

# Test certificate chain
openssl s_client -starttls smtp -connect mail.example.com:25 -showcerts

# Check if intermediate certificates included
sudo postconf smtpd_tls_cert_file
# Should include full chain or use smtpd_tls_chain_files

Solution:

# Option 1: Include chain in certificate file
cat server.crt intermediate.crt > /etc/pki/tls/certs/mail-chain.crt

# Update config
sudo postconf -e 'smtpd_tls_cert_file = /etc/pki/tls/certs/mail-chain.crt'
sudo postfix reload

# Option 2: Use separate chain file (Postfix 3.4+)
sudo postconf -e 'smtpd_tls_chain_files = /etc/pki/tls/certs/chain.crt'

Issue 2: “Permission denied” on Private Key

Symptom: Postfix won’t start, logs show permission error

Fix:

# Set correct permissions
sudo chmod 600 /etc/pki/tls/private/mail.key
sudo chown root:postfix /etc/pki/tls/private/mail.key

# Fix SELinux context
sudo restorecon -v /etc/pki/tls/private/mail.key

# Restart Postfix
sudo systemctl restart postfix

Issue 3: TLS Not Offered to Clients

Symptom: STARTTLS not shown in EHLO response

Diagnosis:

# Telnet to server
telnet mail.example.com 25
# Type: EHLO test
# Should show: 250-STARTTLS

# If not shown, check config
sudo postconf smtpd_tls_security_level
# Should be 'may' or 'encrypt', not 'none'

Solution:

sudo postconf -e 'smtpd_tls_security_level = may'
sudo postfix reload

16.18 Version Comparison Table

16.19 Quick Setup Script

#!/bin/bash
# setup-postfix-tls.sh - Complete Postfix TLS setup

DOMAIN="example.com"
HOSTNAME="mail.$DOMAIN"

echo "=== Postfix TLS Setup for $HOSTNAME ==="

# 1. Install Postfix
sudo dnf install -y postfix

# 2. Generate certificate (or use certmonger)
sudo openssl genpkey -algorithm RSA \
  -out /etc/pki/tls/private/$HOSTNAME.key \
  -pkeyopt rsa_keygen_bits:2048

sudo chmod 600 /etc/pki/tls/private/$HOSTNAME.key

# 3. Generate self-signed for testing (replace with proper cert!)
sudo openssl req -new -x509 -days 365 \
  -key /etc/pki/tls/private/$HOSTNAME.key \
  -out /etc/pki/tls/certs/$HOSTNAME.crt \
  -subj "/CN=$HOSTNAME" \
  -addext "subjectAltName=DNS:$HOSTNAME,DNS:smtp.$DOMAIN"

# 4. Configure Postfix
sudo postconf -e "smtpd_tls_cert_file = /etc/pki/tls/certs/$HOSTNAME.crt"
sudo postconf -e "smtpd_tls_key_file = /etc/pki/tls/private/$HOSTNAME.key"
sudo postconf -e "smtpd_tls_security_level = may"
sudo postconf -e "smtpd_tls_auth_only = yes"
sudo postconf -e "smtpd_tls_loglevel = 1"

sudo postconf -e "smtp_tls_cert_file = /etc/pki/tls/certs/$HOSTNAME.crt"
sudo postconf -e "smtp_tls_key_file = /etc/pki/tls/private/$HOSTNAME.key"
sudo postconf -e "smtp_tls_security_level = may"

# 5. Open firewall
sudo firewall-cmd --permanent --add-service=smtp
sudo firewall-cmd --permanent --add-service=smtps
sudo firewall-cmd --permanent --add-service=smtp-submission
sudo firewall-cmd --reload

# 6. Start Postfix
sudo systemctl enable postfix
sudo systemctl restart postfix

# 7. Test
echo "Testing SMTP STARTTLS..."
echo "QUIT" | openssl s_client -starttls smtp -connect localhost:25 2>&1 | grep -E "(CONNECTED|subject=)"

echo "✅ Postfix TLS setup complete!"
echo "⚠️ Replace self-signed cert with proper certificate from CA"

16.20 Key Takeaways

1. Postfix supports TLS on ports 25 (STARTTLS), 465 (SMTPS), 587 (Submission)
2. RHEL 7 requires manual TLS config (protocols, ciphers)
3. RHEL 8+ uses crypto-policies (much simpler!)
4. Security levels: none, may, encrypt, dane
5. certmonger works great with Postfix
6. Always test with openssl s_client -starttls smtp
7. Monitor logs for TLS errors

Quick Reference Card

┌──────────────────────────────────────────────────────────────┐
│ POSTFIX TLS QUICK REFERENCE                                  │
├──────────────────────────────────────────────────────────────┤
│ Config:       /etc/postfix/main.cf                           │
│ Certs:        smtpd_tls_cert_file = /path/to/cert.crt        │
│ Keys:         smtpd_tls_key_file = /path/to/key.key          │
│ Security:     smtpd_tls_security_level = may|encrypt         │
│                                                              │
│ Test:         openssl s_client -starttls smtp -connect :25   │
│ Reload:       postfix reload                                 │
│ Check:        postfix check                                  │
│ Logs:         tail -f /var/log/maillog | grep TLS            │
│                                                              │
│ Ports:        25 (SMTP+STARTTLS)                             │
│               465 (SMTPS - direct TLS)                       │
│               587 (Submission with STARTTLS)                 │
│                                                              │
│ certmonger:   ipa-getcert ... -C "postfix reload"            │
└──────────────────────────────────────────────────────────────┘

⚠️ RHEL 7: Manual protocol/cipher config needed
✅ RHEL 8/9/10: Crypto-policies handle TLS settings

🧪 Hands-On Lab

Lab 08: Postfix TLS

Configure TLS for Postfix mail server

• 📁 Location: labs/en_US/08-postfix-tls/
• ⏱️ Time: 30 minutes
• 🎯 Level: Intermediate

Chapter Navigation

Chapter 17: OpenLDAP & Directory Services

Enterprise Directory: Learn how to secure OpenLDAP directory services with TLS/SSL on RHEL, protecting user authentication and directory queries.

17.1 LDAP vs LDAPS vs STARTTLS

Three Ways to Secure LDAP

Recommendation: Use LDAPS (port 636) for simplicity and security.

17.2 OpenLDAP Installation

All RHEL Versions

#============================================#
# INSTALL OPENLDAP SERVER
#============================================#

# Install OpenLDAP server
sudo dnf install openldap-servers openldap-clients -y

# Start and enable
sudo systemctl enable slapd
sudo systemctl start slapd

# Open firewall
sudo firewall-cmd --permanent --add-service=ldap
sudo firewall-cmd --permanent --add-service=ldaps
sudo firewall-cmd --reload

# Verify
systemctl status slapd
ss -tlnp | grep -E ':(389|636)'

17.3 Generating Certificates for LDAP

Certificate Requirements

LDAP certificates should include:

• ✅ CN matching LDAP server hostname
• ✅ SANs with LDAP server FQDN
• ✅ Server Authentication key usage
• ✅ Valid trust chain

#============================================#
# GENERATE LDAP SERVER CERTIFICATE
#============================================#

# Step 1: Generate private key
sudo openssl genpkey -algorithm RSA \
  -out /etc/openldap/certs/ldap.key \
  -pkeyopt rsa_keygen_bits:2048

# Step 2: Set permissions (important!)
sudo chmod 600 /etc/openldap/certs/ldap.key
sudo chown ldap:ldap /etc/openldap/certs/ldap.key

# Step 3: Generate CSR
sudo openssl req -new \
  -key /etc/openldap/certs/ldap.key \
  -out /tmp/ldap.csr \
  -subj "/CN=ldap.example.com" \
  -addext "subjectAltName=DNS:ldap.example.com,DNS:dir.example.com"

# Step 4: Get certificate from CA

# Step 5: Install certificate
sudo cp ldap.crt /etc/openldap/certs/
sudo chmod 644 /etc/openldap/certs/ldap.crt
sudo chown ldap:ldap /etc/openldap/certs/ldap.crt

# Step 6: Install CA certificate
sudo cp ca.crt /etc/openldap/certs/
sudo chmod 644 /etc/openldap/certs/ca.crt

17.4 OpenLDAP TLS Configuration

Method 1: cn=config (Dynamic Configuration)

#============================================#
# CONFIGURE TLS WITH cn=config (PREFERRED)
#============================================#

# Create LDIF file
cat > /tmp/tls-config.ldif << EOF
dn: cn=config
changetype: modify
add: olcTLSCertificateFile
olcTLSCertificateFile: /etc/openldap/certs/ldap.crt
-
add: olcTLSCertificateKeyFile
olcTLSCertificateKeyFile: /etc/openldap/certs/ldap.key
-
add: olcTLSCACertificateFile
olcTLSCACertificateFile: /etc/openldap/certs/ca.crt
-
replace: olcTLSProtocolMin
olcTLSProtocolMin: 3.2
EOF

# Apply configuration
sudo ldapmodify -Y EXTERNAL -H ldapi:/// -f /tmp/tls-config.ldif

# Restart slapd
sudo systemctl restart slapd

# Verify
sudo slapcat -b "cn=config" | grep -i tls

Method 2: slapd.conf (Legacy)

#============================================#
# CONFIGURE TLS WITH slapd.conf (LEGACY)
#============================================#

# Edit /etc/openldap/slapd.conf

TLSCertificateFile      /etc/openldap/certs/ldap.crt
TLSCertificateKeyFile   /etc/openldap/certs/ldap.key
TLSCACertificateFile    /etc/openldap/certs/ca.crt

# RHEL 7: Manually specify TLS version
TLSProtocolMin          1.2

# Restart
sudo systemctl restart slapd

17.5 Client Configuration

Configure LDAP Client for TLS

#============================================#
# /etc/openldap/ldap.conf - CLIENT CONFIG
#============================================#

# Server URI (use ldaps:// for port 636)
URI ldaps://ldap.example.com

# CA certificate for validation
TLS_CACERT /etc/pki/tls/certs/ca-bundle.crt

# Certificate verification
TLS_REQCERT demand  # require valid certificate

# RHEL 7: Specify minimum TLS version
# TLS_PROTOCOL_MIN 1.2

Test LDAP Client Connection

#============================================#
# TEST LDAPS CLIENT CONNECTION
#============================================#

# Test LDAPS (port 636)
ldapsearch -H ldaps://ldap.example.com:636 \
  -D "cn=admin,dc=example,dc=com" \
  -W \
  -b "dc=example,dc=com" \
  "(objectClass=*)"

# Test LDAP with STARTTLS (port 389)
ldapsearch -H ldap://ldap.example.com:389 -ZZ \
  -D "cn=admin,dc=example,dc=com" \
  -W \
  -b "dc=example,dc=com"

# -ZZ enforces STARTTLS (fails if unavailable)
# -Z attempts STARTTLS (continues without if unavailable)

17.6 FreeIPA Integration

Note: FreeIPA is covered in detail in Chapter 19. This is a quick overview.

FreeIPA Automatically Handles LDAPS

#============================================#
# FREEIPA LDAPS (AUTOMATIC!)
#============================================#

# FreeIPA automatically configures LDAPS
# No manual certificate config needed!

# Test FreeIPA LDAPS
ldapsearch -H ldaps://ipa.example.com:636 \
  -D "uid=admin,cn=users,cn=accounts,dc=example,dc=com" \
  -W \
  -b "dc=example,dc=com"

# FreeIPA certificates managed by certmonger automatically
sudo getcert list | grep -A10 "Directory Server"

17.7 Testing OpenLDAP TLS

Comprehensive Testing

#============================================#
# OPENLDAP TLS TESTING
#============================================#

# Test 1: Check if slapd is listening
ss -tlnp | grep slapd
# Should show ports 389 and/or 636

# Test 2: Test LDAPS connection with OpenSSL
openssl s_client -connect ldap.example.com:636

# Look for:
# - Successful TLS handshake
# - Certificate details
# - Verify return code: 0 (ok)

# Test 3: Test with ldapsearch (anonymous)
ldapsearch -H ldaps://ldap.example.com:636 \
  -x -b "" -s base "(objectClass=*)" namingContexts

# Test 4: Test authenticated query
ldapsearch -H ldaps://ldap.example.com:636 \
  -D "cn=admin,dc=example,dc=com" \
  -W \
  -b "dc=example,dc=com" \
  "(uid=*)"

# Test 5: Test STARTTLS
ldapsearch -H ldap://ldap.example.com:389 -ZZ \
  -x -b "" -s base

# Test 6: Verify certificate from server
echo | openssl s_client -connect ldap.example.com:636 2>&1 | \
  openssl x509 -noout -subject -issuer -dates

17.8 Troubleshooting OpenLDAP TLS

Diagnostic Commands

#============================================#
# OPENLDAP TLS DIAGNOSTICS
#============================================#

# Check slapd configuration
sudo slapcat -b "cn=config" | grep -i tls

# Check certificate files
sudo ls -lZ /etc/openldap/certs/

# Verify permissions
# Key should be readable by 'ldap' user
sudo -u ldap cat /etc/openldap/certs/ldap.key >/dev/null && \
  echo "✅ Key readable" || echo "❌ Permission denied"

# Check SELinux context
ls -Z /etc/openldap/certs/*.{crt,key}

# Check slapd logs
sudo journalctl -u slapd -f

# Test with verbose OpenSSL
openssl s_client -connect ldap.example.com:636 -showcerts -debug

# Test STARTTLS
ldapsearch -H ldap://ldap.example.com:389 -ZZ -d 1

Common OpenLDAP TLS Issues

17.9 Client Certificate Authentication

Require Client Certificates

#============================================#
# OPENLDAP WITH CLIENT CERT AUTH
#============================================#

# Server configuration (cn=config)
cat > /tmp/client-cert.ldif << EOF
dn: cn=config
changetype: modify
add: olcTLSVerifyClient
olcTLSVerifyClient: demand
-
add: olcTLSCACertificateFile
olcTLSCACertificateFile: /etc/openldap/certs/client-ca.crt
EOF

sudo ldapmodify -Y EXTERNAL -H ldapi:/// -f /tmp/client-cert.ldif

# Restart
sudo systemctl restart slapd

Client connection with certificate:

# Client must provide certificate
ldapsearch -H ldaps://ldap.example.com:636 \
  -x -b "dc=example,dc=com" \
  -ZZ

# Configure client cert in /etc/openldap/ldap.conf:
TLS_CERT /etc/openldap/certs/client.crt
TLS_KEY /etc/openldap/certs/client.key

17.10 Version-Specific Considerations

RHEL 7

#============================================#
# OPENLDAP TLS - RHEL 7
#============================================#

# Manual TLS protocol specification
# In slapd.conf or cn=config:
TLSProtocolMin 1.2

# Or with cn=config:
olcTLSProtocolMin: 3.2  # 3.1=TLS1.0, 3.2=TLS1.1, 3.3=TLS1.2

# Manual cipher configuration
TLSCipherSuite HIGH:!aNULL:!MD5:!3DES

# Test
openssl s_client -connect ldap.example.com:636 -tls1_2

RHEL 8/9/10

#============================================#
# OPENLDAP TLS - RHEL 8/9/10
#============================================#

# Crypto-policies automatically configure TLS
# No need to specify TLSProtocolMin or ciphers!

# Just configure certificates:
olcTLSCertificateFile: /etc/openldap/certs/ldap.crt
olcTLSCertificateKeyFile: /etc/openldap/certs/ldap.key
olcTLSCACertificateFile: /etc/openldap/certs/ca.crt

# Crypto-policy handles the rest
update-crypto-policies --show

17.11 certmonger with OpenLDAP

Automated Certificate Management

#============================================#
# CERTMONGER + OPENLDAP
#============================================#

# Install certmonger
# RHEL 8/9/10
sudo dnf install certmonger

# RHEL 7
# sudo yum install certmonger
sudo systemctl enable --now certmonger

# Request certificate from FreeIPA
sudo ipa-getcert request \
  -f /etc/openldap/certs/ldap.crt \
  -k /etc/openldap/certs/ldap.key \
  -D ldap.example.com \
  -K ldap/ldap.example.com@REALM \
  -C "systemctl restart slapd"  # Restart slapd after renewal

# Set proper ownership
sudo chown ldap:ldap /etc/openldap/certs/ldap.{crt,key}
sudo chmod 600 /etc/openldap/certs/ldap.key

# Monitor
sudo getcert list

17.12 Troubleshooting LDAPS

Diagnostic Steps

#============================================#
# LDAPS TROUBLESHOOTING
#============================================#

# Step 1: Verify slapd is listening on 636
ss -tlnp | grep 636

# Step 2: Check certificate configuration
sudo slapcat -b "cn=config" | grep olcTLS

# Step 3: Test certificate file
sudo openssl x509 -in /etc/openldap/certs/ldap.crt -noout -text

# Step 4: Test key file
sudo openssl rsa -in /etc/openldap/certs/ldap.key -check

# Step 5: Verify cert/key match
CERT_MOD=$(openssl x509 -noout -modulus -in /etc/openldap/certs/ldap.crt | openssl md5)
KEY_MOD=$(openssl rsa -noout -modulus -in /etc/openldap/certs/ldap.key | openssl md5)
[ "$CERT_MOD" = "$KEY_MOD" ] && echo "✅ Match" || echo "❌ Mismatch!"

# Step 6: Check permissions
ls -l /etc/openldap/certs/
# Key should be owned by ldap:ldap with mode 600

# Step 7: Test connection
openssl s_client -connect ldap.example.com:636

# Step 8: Check logs
sudo journalctl -u slapd | grep -i tls

# Step 9: Test from client
ldapsearch -H ldaps://ldap.example.com:636 -x -b "" -s base

# Step 10: Check SELinux
sudo ausearch -m avc -ts recent | grep ldap

17.13 Common Issues and Solutions

Issue 1: “TLS: can’t accept” Error

Symptom: LDAPS connection refused

Diagnosis:

sudo journalctl -u slapd | grep "TLS: can't accept"

Causes & Solutions:

# Cause 1: Key not readable by ldap user
sudo chown ldap:ldap /etc/openldap/certs/ldap.key
sudo chmod 600 /etc/openldap/certs/ldap.key

# Cause 2: SELinux blocking
sudo restorecon -Rv /etc/openldap/certs/
# Or check for denials:
sudo ausearch -m avc -ts recent | grep ldap

# Cause 3: Certificate/key mismatch
# Regenerate CSR with correct key

Issue 2: “TLS: hostname does not match”

Symptom: Client gets hostname mismatch error

Diagnosis:

# Check certificate CN and SANs
openssl x509 -in /etc/openldap/certs/ldap.crt -noout -subject -ext subjectAltName

Solution:

# Reissue certificate with correct hostname in SANs
openssl req -new -key /etc/openldap/certs/ldap.key -out /tmp/ldap.csr \
  -subj "/CN=ldap.example.com" \
  -addext "subjectAltName=DNS:ldap.example.com,DNS:ldap,IP:10.0.0.10"

# Or on client side: allow looser verification (NOT recommended for production)
# In /etc/openldap/ldap.conf:
TLS_REQCERT allow  # instead of 'demand'

Issue 3: Certificate Not Trusted

Symptom: “Certificate verification failed”

Solution:

# Add CA to system trust store
sudo cp ldap-ca.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

# Or specify CA in client config
# /etc/openldap/ldap.conf:
TLS_CACERT /etc/openldap/certs/ca.crt

17.14 Security Best Practices

Hardened OpenLDAP TLS Configuration

#============================================#
# HARDENED LDAP TLS CONFIG (cn=config)
#============================================#

dn: cn=config
changetype: modify
replace: olcTLSCertificateFile
olcTLSCertificateFile: /etc/openldap/certs/ldap.crt
-
replace: olcTLSCertificateKeyFile
olcTLSCertificateKeyFile: /etc/openldap/certs/ldap.key
-
replace: olcTLSCACertificateFile
olcTLSCACertificateFile: /etc/openldap/certs/ca.crt
-
replace: olcTLSProtocolMin
olcTLSProtocolMin: 3.3
-
replace: olcTLSCipherSuite
olcTLSCipherSuite: HIGH:!aNULL:!MD5:!RC4
-
add: olcTLSVerifyClient
olcTLSVerifyClient: never

Apply:

sudo ldapmodify -Y EXTERNAL -H ldapi:/// -f hardened-tls.ldif
sudo systemctl restart slapd

17.15 Integration with System Services

SSSD with LDAPS (System Authentication)

#============================================#
# CONFIGURE SSSD TO USE LDAPS
#============================================#

# /etc/sssd/sssd.conf

[domain/example.com]
id_provider = ldap
auth_provider = ldap
ldap_uri = ldaps://ldap.example.com:636
ldap_search_base = dc=example,dc=com
ldap_tls_cacert = /etc/pki/tls/certs/ca-bundle.crt
ldap_tls_reqcert = demand

# Restart SSSD
sudo systemctl restart sssd

# Test
id ldapuser@example.com

17.16 Monitoring LDAP TLS

Monitoring Commands

#============================================#
# MONITOR LDAP TLS
#============================================#

# Check certificate expiration
openssl s_client -connect ldap.example.com:636 2>/dev/null | \
  openssl x509 -noout -dates

# Check TLS connections
sudo journalctl -u slapd | grep "TLS established"

# Count LDAPS vs LDAP connections
sudo journalctl -u slapd --since today | grep -c "conn=.*LDAPS"
sudo journalctl -u slapd --since today | grep -c "conn=.*LDAP"

# Check for TLS errors
sudo journalctl -u slapd | grep -i "tls.*error"

# certmonger status (if used)
sudo getcert list -d /etc/openldap/certs

17.17 Key Takeaways

1. LDAPS (port 636) is simpler than STARTTLS
2. Certificate ownership critical - Must be readable by ldap user
3. cn=config preferred over slapd.conf (modern RHEL)
4. FreeIPA handles LDAPS automatically - Much easier!
5. Client configuration matters - Set TLS_CACERT correctly
6. Test thoroughly - Use openssl s_client and ldapsearch -ZZ
7. certmonger automates certificate renewal

Quick Reference Card

┌──────────────────────────────────────────────────────────────┐
│ OPENLDAP TLS QUICK REFERENCE                                 │
├──────────────────────────────────────────────────────────────┤
│ Config:       cn=config (dynamic) or slapd.conf (legacy)     │
│ Certs:        /etc/openldap/certs/                           │
│ Ownership:    chown ldap:ldap *.{crt,key}                    │
│ Permissions:  chmod 600 ldap.key                             │
│                                                              │
│ LDAPS:        Port 636 (TLS from start)                      │
│ STARTTLS:     Port 389 (upgrade to TLS)                      │
│                                                              │
│ Test LDAPS:   openssl s_client -connect host:636             │
│ Test STLS:    ldapsearch -H ldap://host:389 -ZZ              │
│                                                              │
│ Client:       /etc/openldap/ldap.conf                        │
│               TLS_CACERT /path/to/ca.crt                     │
│               TLS_REQCERT demand                             │
│                                                              │
│ Logs:         journalctl -u slapd | grep TLS                 │
└──────────────────────────────────────────────────────────────┘

⚠️ Key must be owned by 'ldap' user!
✅ Use FreeIPA for easier LDAP+TLS management

🧪 Hands-On Lab

Lab 09: OpenLDAP LDAPS

Configure LDAPS for secure directory services

• 📁 Location: labs/en_US/09-openldap-ldaps/
• ⏱️ Time: 35-40 minutes
• 🎯 Level: Intermediate

Chapter Navigation

Chapter 18: Database TLS (PostgreSQL, MySQL)

Data in Transit: Protect database connections with TLS encryption. Learn how to configure PostgreSQL and MySQL/MariaDB with certificates on RHEL.

18.1 Why Database TLS?

Protect Sensitive Data:

• ✅ Encrypt database queries and results
• ✅ Prevent eavesdropping on credentials
• ✅ Authenticate database servers
• ✅ Enable client certificate authentication
• ✅ Meet compliance requirements (PCI-DSS, HIPAA)

Threat Model:

• Without TLS: Passwords and data travel in cleartext
• With TLS: All communication encrypted

18.2 PostgreSQL with SSL/TLS

Installation

#============================================#
# INSTALL POSTGRESQL
#============================================#

# RHEL 7/8/9/10
sudo dnf install postgresql-server -y

# Initialize database
sudo postgresql-setup --initdb

# Enable and start
sudo systemctl enable postgresql
sudo systemctl start postgresql

# Verify
systemctl status postgresql
ss -tlnp | grep 5432

Generate PostgreSQL Certificates

#============================================#
# GENERATE POSTGRESQL CERTIFICATES
#============================================#

# Step 1: Generate server key
sudo -u postgres openssl genpkey -algorithm RSA \
  -out /var/lib/pgsql/data/server.key \
  -pkeyopt rsa_keygen_bits:2048

# Step 2: Set permissions (critical!)
sudo chmod 600 /var/lib/pgsql/data/server.key
sudo chown postgres:postgres /var/lib/pgsql/data/server.key

# Step 3: Generate CSR
sudo -u postgres openssl req -new \
  -key /var/lib/pgsql/data/server.key \
  -out /tmp/postgres.csr \
  -subj "/CN=db.example.com" \
  -addext "subjectAltName=DNS:db.example.com,DNS:postgres.example.com"

# Step 4: Get certificate from CA

# Step 5: Install certificate
sudo cp postgres.crt /var/lib/pgsql/data/server.crt
sudo chmod 600 /var/lib/pgsql/data/server.crt
sudo chown postgres:postgres /var/lib/pgsql/data/server.crt

# Step 6: Install CA certificate
sudo cp ca.crt /var/lib/pgsql/data/root.crt
sudo chmod 644 /var/lib/pgsql/data/root.crt

Configure PostgreSQL for SSL

#============================================#
# CONFIGURE POSTGRESQL SSL
#============================================#

# Edit /var/lib/pgsql/data/postgresql.conf
sudo -u postgres vi /var/lib/pgsql/data/postgresql.conf

# Enable SSL
ssl = on

# Certificate files (relative to data directory)
ssl_cert_file = 'server.crt'
ssl_key_file = 'server.key'
ssl_ca_file = 'root.crt'

# RHEL 7: Specify minimum TLS version
# ssl_min_protocol_version = 'TLSv1.2'

# RHEL 8/9/10: Uses system crypto-policy
# (no need to specify ssl_min_protocol_version)

# Optional: Prefer server ciphers
ssl_prefer_server_ciphers = on

# Restart PostgreSQL
sudo systemctl restart postgresql

Configure Client Authentication

#============================================#
# /var/lib/pgsql/data/pg_hba.conf
#============================================#

# Require SSL for all connections
hostssl all all 0.0.0.0/0 md5

# Require client certificate
hostssl all alice 0.0.0.0/0 cert clientcert=verify-full

# Reload configuration
sudo systemctl reload postgresql

HBA Types:

• host: Allow non-SSL
• hostssl: Require SSL
• hostnossl: Explicitly forbid SSL

Client Cert Options:

• md5: SSL required, password auth
• cert: SSL + client certificate required
• clientcert=verify-full: Verify client cert against CA

Test PostgreSQL SSL

#============================================#
# TEST POSTGRESQL SSL
#============================================#

# Test 1: Connect with SSL required
psql "host=db.example.com port=5432 user=testuser dbname=testdb sslmode=require"

# Test 2: Connect with full verification
psql "host=db.example.com port=5432 user=testuser dbname=testdb sslmode=verify-full sslrootcert=/etc/pki/tls/certs/ca-bundle.crt"

# Test 3: With client certificate
psql "host=db.example.com port=5432 user=alice dbname=mydb sslmode=verify-full sslcert=/home/alice/.postgresql/client.crt sslkey=/home/alice/.postgresql/client.key sslrootcert=/etc/pki/tls/certs/ca-bundle.crt"

# Test 4: Check SSL from within PostgreSQL
psql -h db.example.com -U testuser -d testdb -c "SELECT ssl, version FROM pg_stat_ssl WHERE pid = pg_backend_pid();"

# Test 5: OpenSSL test
openssl s_client -connect db.example.com:5432 -starttls postgres

SSL Modes:

• disable: No SSL
• allow: Try SSL, fallback to non-SSL
• prefer: Prefer SSL, fallback allowed
• require: Require SSL (don’t verify cert)
• verify-ca: Require SSL, verify CA
• verify-full: Require SSL, verify hostname + CA

18.3 MySQL/MariaDB with SSL/TLS

Installation

#============================================#
# INSTALL MARIADB (MYSQL REPLACEMENT ON RHEL 8+)
#============================================#

# RHEL 8/9/10
sudo dnf install mariadb-server -y

# Start and enable
sudo systemctl enable mariadb
sudo systemctl start mariadb

# Secure installation
sudo mysql_secure_installation

# Verify
systemctl status mariadb
ss -tlnp | grep 3306

Generate MySQL/MariaDB Certificates

#============================================#
# GENERATE MYSQL/MARIADB CERTIFICATES
#============================================#

# Create certificate directory
sudo mkdir -p /etc/mysql/certs
sudo chmod 755 /etc/mysql/certs

# Step 1: Generate server key
sudo openssl genpkey -algorithm RSA \
  -out /etc/mysql/certs/server.key \
  -pkeyopt rsa_keygen_bits:2048

sudo chmod 600 /etc/mysql/certs/server.key
sudo chown mysql:mysql /etc/mysql/certs/server.key

# Step 2: Generate CSR
sudo openssl req -new \
  -key /etc/mysql/certs/server.key \
  -out /tmp/mysql.csr \
  -subj "/CN=db.example.com" \
  -addext "subjectAltName=DNS:db.example.com,DNS:mysql.example.com"

# Step 3: Get certificate from CA

# Step 4: Install certificate and CA
sudo cp mysql.crt /etc/mysql/certs/server.crt
sudo cp ca.crt /etc/mysql/certs/ca.crt
sudo chmod 644 /etc/mysql/certs/{server.crt,ca.crt}
sudo chown mysql:mysql /etc/mysql/certs/{server.crt,ca.crt}

Configure MySQL/MariaDB for SSL

#============================================#
# CONFIGURE MYSQL/MARIADB SSL
#============================================#

# Edit /etc/my.cnf.d/server.cnf (or /etc/my.cnf)
sudo vi /etc/my.cnf.d/server.cnf

[mysqld]
# SSL/TLS configuration
ssl-ca=/etc/mysql/certs/ca.crt
ssl-cert=/etc/mysql/certs/server.crt
ssl-key=/etc/mysql/certs/server.key

# Require secure transport (optional, forces TLS for all)
require_secure_transport=ON

# RHEL 7: Specify TLS version
# tls_version=TLSv1.2,TLSv1.3

# Restart MySQL/MariaDB
sudo systemctl restart mariadb

Verify SSL is Enabled

#============================================#
# VERIFY MYSQL SSL
#============================================#

# Connect and check SSL status
mysql -u root -p -e "SHOW VARIABLES LIKE '%ssl%';"

# Should show:
# have_ssl           | YES
# ssl_ca             | /etc/mysql/certs/ca.crt
# ssl_cert           | /etc/mysql/certs/server.crt
# ssl_key            | /etc/mysql/certs/server.key

# Check active connections
mysql -u root -p -e "SHOW STATUS LIKE 'Ssl_cipher';"

Test MySQL SSL Connection

#============================================#
# TEST MYSQL SSL CONNECTION
#============================================#

# Connect with SSL
mysql --ssl-ca=/etc/mysql/certs/ca.crt \
  --ssl-mode=REQUIRED \
  -h db.example.com \
  -u testuser \
  -p

# Verify SSL in use
mysql> \s
# Look for: "SSL: Cipher in use is ..."

# Or check from command line
mysql -h db.example.com -u testuser -p -e "STATUS" | grep SSL

18.4 Client Certificate Authentication

PostgreSQL with Client Certificates

#============================================#
# POSTGRESQL CLIENT CERT AUTH
#============================================#

# Server side: pg_hba.conf
hostssl all alice 0.0.0.0/0 cert clientcert=verify-full

# Generate client certificate
openssl genpkey -algorithm RSA -out alice.key -pkeyopt rsa_keygen_bits:2048
openssl req -new -key alice.key -out alice.csr -subj "/CN=alice"
# Get signed by CA

# Client connection
psql "host=db.example.com user=alice dbname=mydb sslmode=verify-full sslcert=alice.crt sslkey=alice.key sslrootcert=ca.crt"

MySQL/MariaDB with Client Certificates

#============================================#
# MYSQL/MARIADB CLIENT CERT AUTH
#============================================#

# Create user requiring X.509
mysql -u root -p << EOF
CREATE USER 'alice'@'%' REQUIRE X509;
GRANT ALL ON mydb.* TO 'alice'@'%';
FLUSH PRIVILEGES;
EOF

# Client connection
mysql --ssl-ca=ca.crt \
  --ssl-cert=alice.crt \
  --ssl-key=alice.key \
  -h db.example.com \
  -u alice \
  -p mydb

18.5 Troubleshooting Database TLS

PostgreSQL Troubleshooting

#============================================#
# POSTGRESQL SSL TROUBLESHOOTING
#============================================#

# Check SSL is enabled
sudo -u postgres psql -c "SHOW ssl;"
# Should show: on

# View SSL settings
sudo -u postgres psql -c "SHOW ssl_cert_file; SHOW ssl_key_file; SHOW ssl_ca_file;"

# Check certificate files
ls -l /var/lib/pgsql/data/server.{crt,key}

# Verify ownership
# Should be: postgres:postgres

# Check permissions
# server.key should be 600

# Test connection with SSL debugging
psql "host=db.example.com sslmode=require" -d postgres -U testuser --set=sslcompression=on

# Check PostgreSQL logs
sudo tail -f /var/lib/pgsql/data/log/postgresql-*.log | grep -i ssl

MySQL/MariaDB Troubleshooting

#============================================#
# MYSQL/MARIADB SSL TROUBLESHOOTING
#============================================#

# Check SSL variables
mysql -u root -p -e "SHOW VARIABLES LIKE '%ssl%';"

# If have_ssl = NO, check:
# 1. Certificate files exist
ls -l /etc/mysql/certs/

# 2. Permissions
# Should be readable by mysql user

# 3. Restart database
sudo systemctl restart mariadb

# Check error log
sudo tail -f /var/log/mariadb/mariadb.log | grep -i ssl

# Test connection
mysql --ssl-ca=/etc/mysql/certs/ca.crt \
  --ssl-mode=REQUIRED \
  -h localhost \
  -u root \
  -p \
  -e "STATUS" | grep SSL

18.6 Common Issues and Solutions

Issue 1: PostgreSQL “Permission denied” on server.key

Symptom: PostgreSQL won’t start, logs show permission error

Fix:

# Set correct permissions
sudo chmod 600 /var/lib/pgsql/data/server.key
sudo chown postgres:postgres /var/lib/pgsql/data/server.key

# Fix SELinux context
sudo restorecon -Rv /var/lib/pgsql/data/

# Restart
sudo systemctl restart postgresql

Issue 2: MySQL “SSL connection error”

Diagnosis:

# Check if SSL is available
mysql -u root -p -e "SHOW VARIABLES LIKE 'have_ssl';"
# Should show: YES

# If shows: DISABLED
# Check certificate paths in my.cnf

Fix:

# Verify paths in /etc/my.cnf.d/server.cnf
[mysqld]
ssl-ca=/etc/mysql/certs/ca.crt
ssl-cert=/etc/mysql/certs/server.crt
ssl-key=/etc/mysql/certs/server.key

# Restart
sudo systemctl restart mariadb

Issue 3: Client Certificate Rejected

Symptom: Connection fails with client cert

PostgreSQL Fix:

# Check pg_hba.conf
cat /var/lib/pgsql/data/pg_hba.conf | grep hostssl

# Ensure client CA is installed
sudo cp client-ca.crt /var/lib/pgsql/data/root.crt

# Reload
sudo systemctl reload postgresql

MySQL Fix:

# Verify user requires X.509
mysql -u root -p -e "SELECT user, host, ssl_type FROM mysql.user WHERE user='alice';"
# Should show: X509

# Verify CA file configured
mysql -u root -p -e "SHOW VARIABLES LIKE 'ssl_ca';"

18.7 Version-Specific Considerations

PostgreSQL Versions on RHEL

MySQL/MariaDB Versions on RHEL

18.8 Performance Considerations

PostgreSQL SSL Performance

#============================================#
# POSTGRESQL SSL PERFORMANCE TUNING
#============================================#

# /var/lib/pgsql/data/postgresql.conf

# SSL enabled
ssl = on

# SSL compression (disabled for security, CRIME attack)
ssl_compression = off

# SSL ciphers (RHEL 7 - manual)
# ssl_ciphers = 'HIGH:!aNULL:!MD5'

# RHEL 8/9/10: crypto-policy handles ciphers

# Connection pooling helps (use pgBouncer)
# SSL termination at proxy can improve performance

MySQL SSL Performance

#============================================#
# MYSQL SSL PERFORMANCE
#============================================#

# [mysqld] in /etc/my.cnf.d/server.cnf

# SSL enabled
ssl-ca=/etc/mysql/certs/ca.crt
ssl-cert=/etc/mysql/certs/server.crt
ssl-key=/etc/mysql/certs/server.key

# Disable weak ciphers (RHEL 7)
# tls_version=TLSv1.2,TLSv1.3
# ssl_cipher='ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256'

# RHEL 8/9/10: crypto-policy handles it

# Connection pooling (use ProxySQL or similar)

18.9 Monitoring Database TLS

PostgreSQL Monitoring

#============================================#
# MONITOR POSTGRESQL SSL
#============================================#

# Check SSL connections
sudo -u postgres psql -c "SELECT datname, usename, ssl, version FROM pg_stat_ssl JOIN pg_stat_activity ON pg_stat_ssl.pid = pg_stat_activity.pid;"

# Count SSL vs non-SSL
sudo -u postgres psql -c "SELECT ssl, COUNT(*) FROM pg_stat_ssl GROUP BY ssl;"

# Check certificate expiration
openssl x509 -in /var/lib/pgsql/data/server.crt -noout -checkend $((86400*30))

# Monitor connections
sudo -u postgres psql -c "SELECT COUNT(*) FROM pg_stat_activity WHERE ssl = true;"

MySQL Monitoring

#============================================#
# MONITOR MYSQL SSL
#============================================#

# Check SSL status
mysql -u root -p -e "SHOW STATUS LIKE 'Ssl%';"

# Count SSL connections
mysql -u root -p -e "SHOW STATUS LIKE 'Ssl_accepts';"

# Current SSL connections
mysql -u root -p -e "SELECT user, host, connection_type FROM information_schema.processlist WHERE connection_type = 'SSL/TLS';"

# Certificate expiration
openssl x509 -in /etc/mysql/certs/server.crt -noout -checkend $((86400*30))

18.10 Complete Setup Scripts

PostgreSQL SSL Setup Script

#!/bin/bash
# setup-postgresql-ssl.sh

echo "=== PostgreSQL SSL Setup ==="

# Generate self-signed cert (replace with proper CA cert!)
sudo -u postgres openssl req -new -x509 -days 365 -nodes \
  -out /var/lib/pgsql/data/server.crt \
  -keyout /var/lib/pgsql/data/server.key \
  -subj "/CN=$(hostname -f)"

# Set permissions
sudo chmod 600 /var/lib/pgsql/data/server.{crt,key}
sudo chown postgres:postgres /var/lib/pgsql/data/server.{crt,key}

# Enable SSL in postgresql.conf
sudo -u postgres psql -c "ALTER SYSTEM SET ssl = on;"

# Restart
sudo systemctl restart postgresql

# Test
sudo -u postgres psql -c "SHOW ssl;"

echo "✅ PostgreSQL SSL enabled"
echo "⚠️ Replace self-signed cert with proper certificate from CA"

18.11 Key Takeaways

1. Both PostgreSQL and MySQL support SSL/TLS
2. File ownership critical - postgres:postgres or mysql:mysql
3. Permissions: 600 for keys, 644 for certs
4. pg_hba.conf controls PostgreSQL access (hostssl)
5. sslmode important for PostgreSQL clients
6. Client certificates enable strong authentication
7. Test thoroughly before enforcing TLS
8. Monitor SSL usage - Ensure clients actually use it

Quick Reference Card

┌──────────────────────────────────────────────────────────────┐
│ DATABASE TLS QUICK REFERENCE                                 │
├──────────────────────────────────────────────────────────────┤
│ === POSTGRESQL ===                                           │
│ Config:       /var/lib/pgsql/data/postgresql.conf            │
│ Access:       /var/lib/pgsql/data/pg_hba.conf                │
│ Certs:        /var/lib/pgsql/data/server.{crt,key}           │
│ Owner:        postgres:postgres                              │
│ Enable:       ssl = on                                       │
│ Test:         psql "sslmode=require"                         │
│                                                              │
│ === MYSQL/MARIADB ===                                        │
│ Config:       /etc/my.cnf.d/server.cnf                       │
│ Certs:        /etc/mysql/certs/server.{crt,key}              │
│ Owner:        mysql:mysql                                    │
│ Enable:       ssl-ca, ssl-cert, ssl-key in [mysqld]          │
│ Test:         mysql --ssl-mode=REQUIRED                      │
│                                                              │
│ Permissions:  chmod 600 *.key                                │
│               chmod 644 *.crt                                │
└──────────────────────────────────────────────────────────────┘

⚠️ File ownership and permissions are critical!
✅ Use hostssl in pg_hba.conf to require SSL

🧪 Hands-On Lab

Lab 10: PostgreSQL TLS

Configure TLS for PostgreSQL database connections

• 📁 Location: labs/en_US/10-postgresql-tls/
• ⏱️ Time: 25-30 minutes
• 🎯 Level: Intermediate

Chapter Navigation

Chapter 19: FreeIPA Certificate Services

Enterprise CA: FreeIPA is Red Hat’s integrated identity and certificate management solution. It’s the recommended way to run an internal CA on RHEL.

19.1 What is FreeIPA?

FreeIPA (Identity, Policy, Audit) is an integrated security information management solution combining:

• 🔐 Identity Management (LDAP directory)
• 🎫 Authentication (Kerberos)
• 🔏 Certificate Authority (Dogtag PKI)
• 📋 Policy Management (sudo, HBAC)
• 🔍 DNS (BIND integration)

Why FreeIPA for Certificates?

Instead of:

• ❌ Manually managing certificates per server
• ❌ External CA costs
• ❌ Complex PKI infrastructure

FreeIPA Provides:

• ✅ Internal CA (free!)
• ✅ Automatic certificate enrollment
• ✅ Auto-renewal via certmonger
• ✅ Certificate profiles
• ✅ Web UI and CLI management
• ✅ Integration with RHEL services

19.2 FreeIPA Installation (Server)

Prerequisites

#============================================#
# PREREQUISITES FOR FREEIPA SERVER
#============================================#

# Requirements:
# - RHEL 7/8/9/10
# - Minimum 2 GB RAM (4 GB recommended)
# - Fully qualified hostname
# - Proper DNS resolution
# - Static IP address

# Verify hostname
hostnamectl
# Must show FQDN: ipa.example.com

# Verify DNS
nslookup $(hostname -f)

# Set hostname if needed
sudo hostnamectl set-hostname ipa.example.com

Install FreeIPA Server

#============================================#
# INSTALL FREEIPA SERVER
#============================================#

# Install packages
sudo dnf install ipa-server ipa-server-dns -y

# Run installation wizard
sudo ipa-server-install \
  --realm EXAMPLE.COM \
  --domain example.com \
  --ds-password 'DirectoryPassword123!' \
  --admin-password 'AdminPassword123!' \
  --hostname ipa.example.com \
  --setup-dns \
  --forwarder 8.8.8.8 \
  --forwarder 8.8.4.4 \
  --unattended

# Installation takes 5-15 minutes

# Open firewall
sudo firewall-cmd --add-service={http,https,dns,ntp,freeipa-ldap,freeipa-ldaps,freeipa-replication} --permanent
sudo firewall-cmd --reload

# Verify
sudo ipactl status

# Should show multiple services running:
# - Directory Service (389-ds)
# - Certificate Authority (pki-tomcatd)
# - Kerberos KDC
# - Apache Web Server
# - DNS (named)

Access FreeIPA Web UI

# Get Kerberos ticket
kinit admin
# Password: AdminPassword123!

# Access Web UI
# https://ipa.example.com/
# Username: admin
# Password: AdminPassword123!

19.3 Enrolling Clients

Client Installation

#============================================#
# ENROLL CLIENT TO FREEIPA
#============================================#

# On client system (web01.example.com)

# Install IPA client
sudo dnf install ipa-client -y

# Enroll
sudo ipa-client-install \
  --domain example.com \
  --realm EXAMPLE.COM \
  --server ipa.example.com \
  --principal admin \
  --password 'AdminPassword123!' \
  --mkhomedir \
  --unattended

# Verify
sudo ipa-client-install --uninstall  # Just kidding, don't run this!

# Verify enrollment
ipa ping
# Pong!

# Check certificate tracking
sudo getcert list
# Shows certmonger tracking host certificate

19.4 Requesting Certificates from FreeIPA

Method 1: Web UI

1. Navigate to https://ipa.example.com/
2. Identity → Hosts → Select host → Actions → New Certificate
3. Or: Identity → Services → Add service → Request certificate

Method 2: CLI (Recommended)

#============================================#
# REQUEST CERTIFICATE FROM FREEIPA
#============================================#

# For HTTP service on web01
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/web01.crt \
  -k /etc/pki/tls/private/web01.key \
  -K HTTP/web01.example.com@EXAMPLE.COM \
  -D web01.example.com \
  -C "systemctl reload httpd"

# For custom service
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/myapp.crt \
  -k /etc/pki/tls/private/myapp.key \
  -K myapp/web01.example.com@EXAMPLE.COM \
  -D myapp.example.com

# Check status
sudo getcert list

# Wait for MONITORING status (cert issued)

Method 3: ipa cert-request (Advanced)

#============================================#
# ADVANCED: IPA CERT-REQUEST
#============================================#

# Generate CSR
openssl req -new -key server.key -out server.csr \
  -subj "/CN=server.example.com"

# Request certificate via IPA
ipa cert-request server.csr \
  --principal HTTP/server.example.com@EXAMPLE.COM

# Get certificate ID from output
# Certificate: MIIDXTCCAkWgAwIBAgI...
# Request ID: 12345

# Retrieve certificate
ipa cert-show 12345 --out server.crt

19.5 Certificate Profiles

Available Profiles

#============================================#
# FREEIPA CERTIFICATE PROFILES
#============================================#

# List available profiles
ipa certprofile-find

# Common profiles:
# - caIPAserviceCert: Service certificates (HTTP, LDAP, etc.)
# - IECUserRoles: User certificates
# - smimeUserCert: S/MIME email certificates
# - caSelfSignedCert: Self-signed CA

# View profile details
ipa certprofile-show caIPAserviceCert

# Create custom profile
ipa certprofile-import MyCustomProfile \
  --file custom-profile.cfg \
  --store TRUE

Using Specific Profile

# Request with specific profile
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/custom.crt \
  -k /etc/pki/tls/private/custom.key \
  -K HTTP/web01.example.com@EXAMPLE.COM \
  -T caIPAserviceCert  # Specify profile

19.6 Automatic Renewal

How It Works

FreeIPA + certmonger = Automatic Certificate Lifecycle!

#============================================#
# AUTOMATIC RENEWAL WITH FREEIPA
#============================================#

# certmonger automatically:
# 1. Tracks certificate expiration
# 2. Submits renewal request to IPA
# 3. Obtains renewed certificate
# 4. Saves to file
# 5. Runs post-save command (e.g., reload httpd)

# Check renewal status
sudo getcert list

# Example output:
# Request ID '20240101000000':
#   status: MONITORING
#   stuck: no
#   key pair storage: type=FILE,location='/etc/pki/tls/private/web.key'
#   certificate: type=FILE,location='/etc/pki/tls/certs/web.crt'
#   CA: IPA
#   issuer: CN=Certificate Authority,O=EXAMPLE.COM
#   subject: CN=web01.example.com,O=EXAMPLE.COM
#   expires: 2025-01-01 00:00:00 UTC
#   pre-save command:
#   post-save command: systemctl reload httpd
#   track: yes
#   auto-renew: yes

# Renewal happens automatically ~28 days before expiry!

Manual Renewal (If Needed)

# Force renewal now
sudo ipa-getcert resubmit -f /etc/pki/tls/certs/web.crt

# Or by request ID
sudo ipa-getcert resubmit -i 20240101000000

# Check if successful
sudo getcert list -f /etc/pki/tls/certs/web.crt

19.7 FreeIPA as Enterprise CA

CA Certificate Management

#============================================#
# FREEIPA CA MANAGEMENT
#============================================#

# View CA certificate
ipa ca-show ipa

# Export CA certificate
ipa ca-show ipa --certificate --out /tmp/ipa-ca.crt

# Install on clients (automatic during ipa-client-install)
# Manual: Copy to trust store
sudo cp /tmp/ipa-ca.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

# Renew CA certificate (when needed)
sudo ipa-cacert-manage renew

# Check CA expiration
sudo getcert list -d /var/lib/ipa | grep "CA:"

Sub-CAs (Advanced)

#============================================#
# FREEIPA SUB-CA (RHEL 8+)
#============================================#

# Create sub-CA
ipa ca-add subca \
  --subject "CN=SubCA,O=EXAMPLE.COM" \
  --desc "Department Sub-CA"

# Issue certificate from sub-CA
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/dept.crt \
  -k /etc/pki/tls/private/dept.key \
  -X subca \
  -K HTTP/dept.example.com@EXAMPLE.COM

19.8 Service Integration Examples

Apache with FreeIPA Certificates

#============================================#
# APACHE + FREEIPA COMPLETE SETUP
#============================================#

# 1. Enroll system to IPA (if not already)
sudo ipa-client-install

# 2. Request certificate for Apache
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/$(hostname -f).crt \
  -k /etc/pki/tls/private/$(hostname -f).key \
  -K HTTP/$(hostname -f)@EXAMPLE.COM \
  -D $(hostname -f) \
  -C "systemctl reload httpd"

# 3. Wait for certificate
until sudo getcert list -f /etc/pki/tls/certs/$(hostname -f).crt | grep -q "MONITORING"; do
  sleep 5
  echo "Waiting for certificate..."
done

# 4. Configure Apache to use it
# /etc/httpd/conf.d/ssl.conf:
# SSLCertificateFile /etc/pki/tls/certs/$(hostname -f).crt
# SSLCertificateKeyFile /etc/pki/tls/private/$(hostname -f).key

# 5. Reload Apache
sudo systemctl reload httpd

# Certificate auto-renews!

LDAP with FreeIPA Certificates

# FreeIPA's own LDAP service automatically uses IPA certificates
# No manual configuration needed!

# Test
ldapsearch -H ldaps://ipa.example.com:636 -x -b "dc=example,dc=com"

19.9 Troubleshooting FreeIPA Certificates

Common Issues

Issue 1: CA_UNREACHABLE

# Symptom
sudo getcert list
# status: CA_UNREACHABLE

# Diagnosis
# 1. Check IPA server connectivity
ipa ping

# 2. Check Kerberos ticket
klist

# 3. Renew ticket if expired
kinit -k host/$(hostname -f)@EXAMPLE.COM

# 4. Check IPA services
ssh ipa.example.com "sudo ipactl status"

# 5. Retry
sudo ipa-getcert resubmit -i <request-id>

Issue 2: Certificate Request Denied

# Check request status
sudo getcert list -v

# Common causes:
# 1. Service principal doesn't exist
ipa service-find HTTP/$(hostname -f)

# If not found, add it:
ipa service-add HTTP/$(hostname -f)

# 2. Host not enrolled
ipa host-show $(hostname -f)

# 3. Insufficient permissions
# Must request as enrolled host principal

Issue 3: Certificate Not Renewed

# Check certmonger logs
sudo journalctl -u certmonger | tail -50

# Check IPA CA status
sudo ipactl status | grep "CA"

# Force renewal
sudo ipa-getcert resubmit -f /etc/pki/tls/certs/web.crt

# Check CA certificate expiration
sudo openssl x509 -in /etc/ipa/ca.crt -noout -dates

19.10 Advanced Features

IdM ACME Support (RHEL 9+)

FreeIPA can expose its own internal ACME server. This is not Let’s Encrypt.

#============================================#
# ENABLE ACME ON FREEIPA (RHEL 9+)
#============================================#

# On IPA server (RHEL 9+)
sudo ipa-acme-manage enable

# Verify ACME is available
curl https://ipa.example.com/acme/directory

# On client: Use certbot or another ACME client against the IPA ACME directory
sudo certbot register --server https://ipa.example.com/acme/directory
sudo certbot certonly --server https://ipa.example.com/acme/directory \
  -d web01.example.com

# This uses your IdM / FreeIPA CA, not the public Let's Encrypt service

Certificate Hold/Revocation

#============================================#
# REVOKE CERTIFICATES
#============================================#

# Put certificate on hold (temporary)
ipa cert-revoke 12345 --revocation-reason 6

# Revoke permanently
ipa cert-revoke 12345 --revocation-reason 1

# Reasons:
# 0: unspecified
# 1: keyCompromise
# 2: cACompromise
# 4: superseded
# 6: certificateHold (can be removed)

# Remove from hold
ipa cert-remove-hold 12345

# Check revocation status
ipa cert-show 12345

19.11 Monitoring FreeIPA PKI

Health Checks

#============================================#
# FREEIPA PKI HEALTH MONITORING
#============================================#

# Check IPA overall status
sudo ipactl status

# Check CA subsystem
sudo systemctl status pki-tomcatd@pki-tomcat

# Check certificate expirations
ipa-healthcheck --source ipahealthcheck.ipa.certs

# Check certificate tracking
sudo getcert list | grep -E "(Request|status|expires)"

# Monitor CA certificate
openssl x509 -in /etc/ipa/ca.crt -noout -dates

# Check for expiring certificates
ipa cert-find --validnotafter-from=$(date -d '+60 days' +%Y-%m-%d)

19.12 Backup and Recovery

Backup IPA Server

#============================================#
# BACKUP FREEIPA (INCLUDING CA)
#============================================#

# Full backup
sudo ipa-backup --data --online

# Backup location
ls -lh /var/lib/ipa/backup/

# Include CA keys (offline backup only!)
sudo ipactl stop
sudo ipa-backup --data --gpg
# Enter GPG passphrase
sudo ipactl start

Restore IPA Server

# Restore from backup
sudo ipactl stop
sudo ipa-restore /var/lib/ipa/backup/ipa-full-YYYY-MM-DD-HH-MM-SS/
sudo ipactl start

19.13 Best Practices

FreeIPA Certificate Best Practices

✅ Use FreeIPA for all internal certificates
✅ Let certmonger handle renewal (don't manually renew)
✅ Use service principals (HTTP/host, ldap/host, etc.)
✅ Add SANs when requesting certificates
✅ Set post-save commands (-C flag) for service reload
✅ Monitor IPA server health regularly
✅ Backup IPA server weekly (including CA keys)
✅ Have at least 2 IPA replicas (HA)
✅ Monitor CA certificate expiration
✅ Test certificate renewal before expiry
✅ Use certificate profiles for standardization

19.14 Integration Examples

Complete Service Setup with FreeIPA

#!/bin/bash
# setup-service-with-ipa.sh
# Complete workflow for service certificate from FreeIPA

SERVICE_NAME="HTTP"  # Or LDAP, postgresql, etc.
HOST=$(hostname -f)
PRINCIPAL="${SERVICE_NAME}/${HOST}@EXAMPLE.COM"
CERT_FILE="/etc/pki/tls/certs/${HOST}.crt"
KEY_FILE="/etc/pki/tls/private/${HOST}.key"
POST_COMMAND="systemctl reload httpd"

echo "=== Requesting Certificate from FreeIPA ==="

# 1. Ensure service principal exists
if ! ipa service-show "${SERVICE_NAME}/${HOST}" &>/dev/null; then
  echo "Creating service principal..."
  ipa service-add "${SERVICE_NAME}/${HOST}"
fi

# 2. Request certificate
sudo ipa-getcert request \
  -f "$CERT_FILE" \
  -k "$KEY_FILE" \
  -K "$PRINCIPAL" \
  -D "$HOST" \
  -C "$POST_COMMAND"

# 3. Wait for certificate
echo "Waiting for certificate issuance..."
until sudo getcert list -f "$CERT_FILE" | grep -q "MONITORING"; do
  sleep 5
done

# 4. Verify
echo "✅ Certificate issued!"
sudo openssl x509 -in "$CERT_FILE" -noout -subject -issuer -dates

# 5. Certificate will auto-renew!
echo "✅ Certificate tracking enabled - auto-renewal active"

19.15 Key Takeaways

1. FreeIPA is Red Hat’s recommended internal CA
2. Combines identity + certificates + authentication
3. certmonger integration is automatic
4. Certificates auto-renew (no manual work!)
5. Use service principals (HTTP/host, ldap/host)
6. IdM ACME can expose an internal ACME endpoint in RHEL 9+; keep it distinct from Let’s Encrypt
7. Web UI and CLI both available
8. Scales to enterprise - Supports replicas, sub-CAs

Quick Reference Card

┌──────────────────────────────────────────────────────────────┐
│ FREEIPA CERTIFICATE SERVICES QUICK REFERENCE                 │
├──────────────────────────────────────────────────────────────┤
│ Install:       dnf install ipa-server                        │
│ Setup:         ipa-server-install                            │
│ Status:        ipactl status                                 │
│ Web UI:        https://ipa.example.com/                      │
│                                                              │
│ Enroll:        ipa-client-install                            │
│ Request:       ipa-getcert request -K service/host@REALM     │
│ List:          getcert list                                  │
│ Resubmit:      ipa-getcert resubmit -f /path/to/cert.crt     │
│                                                              │
│ Principal:     HTTP/host.example.com@REALM                   │
│                ldap/host.example.com@REALM                   │
│                postgresql/host.example.com@REALM             │
│                                                              │
│ Auto-renewal:  Automatic via certmonger                      │
│ ACME:          ipa-acme-manage enable (RHEL 9+)              │
└──────────────────────────────────────────────────────────────┘

✅ Best for internal enterprise certificate management
✅ Fully integrated with RHEL
✅ No manual renewal needed!

Chapter Navigation

Chapter 20: Other RHEL Services with Certificates

Beyond the Basics: Many other RHEL services use certificates. This chapter covers Cockpit, VPN services, container registries, and more.

20.1 Services Covered

This chapter provides quick-start guides for:

• 🖥️ Cockpit (Web-based admin console)
• 🔒 OpenVPN (VPN service)
• 🛡️ strongSwan (IPsec VPN)
• 📦 Container Registry (Podman/Docker registry)
• 📡 HAProxy (Load balancer)
• 🔌 Redis with TLS (using stunnel)
• ⚙️ Ansible Tower/AWX (Automation platform)

20.2 Cockpit Web Console

What is Cockpit?

Cockpit is RHEL’s built-in web-based administration interface.

Default: Uses self-signed certificate Goal: Replace with proper certificate

Configure Cockpit with Certificates

#============================================#
# COCKPIT WITH PROPER CERTIFICATE
#============================================#

# Install Cockpit
sudo dnf install cockpit -y
sudo systemctl enable --now cockpit.socket

# Open firewall
sudo firewall-cmd --add-service=cockpit --permanent
sudo firewall-cmd --reload

# Cockpit certificate location
ls -l /etc/cockpit/ws-certs.d/

# Method 1: Place certificate with specific name
# Cockpit uses certificates in /etc/cockpit/ws-certs.d/
# Filename format: NN-name.cert (where NN = priority, lower = higher priority)

sudo cat server.crt server.key > /etc/cockpit/ws-certs.d/01-server.cert
sudo chmod 644 /etc/cockpit/ws-certs.d/01-server.cert

# Restart Cockpit
sudo systemctl restart cockpit.socket

# Method 2: Use certmonger
sudo ipa-getcert request \
  -f /etc/cockpit/ws-certs.d/01-cockpit.cert \
  -k /etc/cockpit/ws-certs.d/01-cockpit.cert \
  -K HTTP/$(hostname -f)@REALM \
  -D $(hostname -f) \
  -C "systemctl restart cockpit.socket" \
  -F /etc/cockpit/ws-certs.d/01-cockpit.cert  # Combined cert+key

# Access Cockpit
# https://server.example.com:9090/

Note: Cockpit expects combined cert+key in single file!

20.3 OpenVPN

Server Configuration with Certificates

#============================================#
# OPENVPN SERVER WITH CERTIFICATES
#============================================#

# Install OpenVPN (from EPEL on RHEL 7, repos on RHEL 8+)
sudo dnf install openvpn -y

# Generate or obtain certificates:
# - CA certificate
# - Server certificate + key
# - Client certificates

# /etc/openvpn/server/server.conf
port 1194
proto udp
dev tun

ca /etc/openvpn/server/ca.crt
cert /etc/openvpn/server/server.crt
key /etc/openvpn/server/server.key
dh /etc/openvpn/server/dh2048.pem

tls-auth /etc/openvpn/server/ta.key 0
cipher AES-256-GCM
auth SHA256

server 10.8.0.0 255.255.255.0

# Start OpenVPN
sudo systemctl enable --now openvpn-server@server

# Open firewall
sudo firewall-cmd --add-port=1194/udp --permanent
sudo firewall-cmd --reload

Test OpenVPN

# Check if running
systemctl status openvpn-server@server

# Test from client
openvpn --config client.ovpn --verb 3

20.4 strongSwan IPsec VPN

Configure with Certificates

#============================================#
# STRONGSWAN WITH CERTIFICATES
#============================================#

# Install
sudo dnf install strongswan -y

# Certificate locations
# CA: /etc/strongswan/ipsec.d/cacerts/
# Server/Client certs: /etc/strongswan/ipsec.d/certs/
# Private keys: /etc/strongswan/ipsec.d/private/

# Copy certificates
sudo cp ca.crt /etc/strongswan/ipsec.d/cacerts/
sudo cp server.crt /etc/strongswan/ipsec.d/certs/
sudo cp server.key /etc/strongswan/ipsec.d/private/
sudo chmod 600 /etc/strongswan/ipsec.d/private/server.key

# /etc/strongswan/ipsec.conf
config setup
    charondebug="ike 2, knl 2, cfg 2"

conn example-ipsec
    left=%any
    leftid=@server.example.com
    leftcert=server.crt
    leftsubnet=10.0.0.0/24

    right=%any
    rightid=@client.example.com
    rightcert=client.crt

    auto=add
    type=tunnel
    keyexchange=ikev2

# Start strongSwan
sudo systemctl enable --now strongswan

# Check status
sudo swanctl --list-sas

20.5 Container Registry with TLS

Podman/Docker Registry

#============================================#
# CONTAINER REGISTRY WITH TLS
#============================================#

# Install registry
sudo dnf install -y podman
sudo podman pull docker.io/library/registry:2

# Create certificate for registry
sudo mkdir -p /etc/registry/certs
sudo openssl genpkey -algorithm RSA \
  -out /etc/registry/certs/registry.key \
  -pkeyopt rsa_keygen_bits:2048

sudo openssl req -new -x509 -days 365 \
  -key /etc/registry/certs/registry.key \
  -out /etc/registry/certs/registry.crt \
  -subj "/CN=registry.example.com" \
  -addext "subjectAltName=DNS:registry.example.com"

# Run registry with TLS
sudo podman run -d \
  --name registry \
  -p 5000:5000 \
  -v /etc/registry/certs:/certs:ro \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/registry.crt \
  -e REGISTRY_HTTP_TLS_KEY=/certs/registry.key \
  registry:2

# Test
curl https://registry.example.com:5000/v2/_catalog

Client Configuration

# Add registry CA to system trust
sudo cp /etc/registry/certs/registry.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

# Test pull
podman pull registry.example.com:5000/myimage:latest

20.6 HAProxy TLS Termination

Load Balancer with TLS

#============================================#
# HAPROXY TLS TERMINATION
#============================================#

# Install HAProxy
sudo dnf install haproxy -y

# HAProxy requires combined cert+key+chain in ONE file
cat server.crt intermediate.crt server.key > /etc/haproxy/certs/bundle.pem
sudo chmod 600 /etc/haproxy/certs/bundle.pem

# /etc/haproxy/haproxy.cfg
frontend https-in
    bind *:443 ssl crt /etc/haproxy/certs/bundle.pem
    mode http
    default_backend web_servers

    # Force HTTPS
    http-request redirect scheme https unless { ssl_fc }

    # Security headers
    http-response set-header Strict-Transport-Security "max-age=31536000"

backend web_servers
    mode http
    balance roundrobin
    server web1 10.0.1.10:80 check
    server web2 10.0.1.11:80 check
    server web3 10.0.1.12:80 check

# Start HAProxy
sudo systemctl enable --now haproxy

# Test
curl -v https://loadbalancer.example.com/

20.7 Redis with TLS (via stunnel)

Redis TLS Proxy

#============================================#
# REDIS WITH TLS USING STUNNEL
#============================================#

# Install Redis and stunnel
sudo dnf install redis stunnel -y

# Configure Redis (listen on localhost only)
# /etc/redis/redis.conf
bind 127.0.0.1

# Start Redis
sudo systemctl enable --now redis

# Configure stunnel
# /etc/stunnel/redis.conf
[redis-tls]
accept = 0.0.0.0:6380
connect = 127.0.0.1:6379
cert = /etc/pki/tls/certs/redis.crt
key = /etc/pki/tls/private/redis.key
CAfile = /etc/pki/tls/certs/ca-bundle.crt

# Optional: Require client certificates
verify = 2
CApath = /etc/pki/tls/certs/

# Start stunnel
sudo systemctl enable --now stunnel@redis

# Test
openssl s_client -connect localhost:6380
# Then type: PING
# Should respond: +PONG

20.8 Ansible Tower/AWX

Tower/AWX with Custom Certificate

#============================================#
# ANSIBLE TOWER/AWX CUSTOM CERTIFICATE
#============================================#

# Tower certificate location
# /etc/tower/tower.cert
# /etc/tower/tower.key

# Replace with proper certificate
sudo cp tower.example.com.crt /etc/tower/tower.cert
sudo cp tower.example.com.key /etc/tower/tower.key
sudo chmod 600 /etc/tower/tower.key

# Restart Tower services
sudo ansible-tower-service restart

# Or for AWX (containerized)
# Update docker-compose.yml or k8s secrets

# Test
curl -v https://tower.example.com/

20.9 SSH with Certificates (Advanced)

SSH Certificate Authentication

Note: Different from SSH keys! This uses X.509 certificates.

#============================================#
# SSH WITH X.509 CERTIFICATES (ADVANCED)
#============================================#

# Requires: openssh-server with X.509 patch or ssh-keysign

# Generate certificate for SSH user
openssl genpkey -algorithm RSA -out ssh-user.key -pkeyopt rsa_keygen_bits:2048
openssl req -new -key ssh-user.key -out ssh-user.csr -subj "/CN=user@example.com"
# Get signed by CA

# Configure sshd (experimental, not standard RHEL)
# /etc/ssh/sshd_config
# X509KeyAlgorithm x509v3-rsa2048-sha256
# X509TrustAnchor /etc/ssh/ca.crt

# Standard approach: Use SSH keys, not X.509
# X.509 SSH support is limited on RHEL

Recommendation: Use standard SSH key-based authentication for SSH, use X.509 for other services.

20.10 Monitoring Multiple Services

Multi-Service Certificate Check

#!/bin/bash
# check-all-services.sh
# Check certificates for all services

echo "=== Multi-Service Certificate Check ==="

# Apache
echo "1. Apache (port 443):"
timeout 3 openssl s_client -connect localhost:443 </dev/null 2>&1 | grep -E "(subject=|issuer=)" | head -2

# NGINX (if installed)
echo "2. NGINX (port 8443 or custom):"
timeout 3 openssl s_client -connect localhost:8443 </dev/null 2>&1 | grep -E "(subject=|issuer=)" | head -2

# Postfix
echo "3. Postfix SMTPS (port 465):"
timeout 3 openssl s_client -connect localhost:465 </dev/null 2>&1 | grep -E "(subject=|issuer=)" | head -2

# LDAPS
echo "4. LDAP (port 636):"
timeout 3 openssl s_client -connect localhost:636 </dev/null 2>&1 | grep -E "(subject=|issuer=)" | head -2

# PostgreSQL
echo "5. PostgreSQL (port 5432):"
sudo -u postgres psql -c "SHOW ssl;" 2>/dev/null

# Cockpit
echo "6. Cockpit (port 9090):"
timeout 3 openssl s_client -connect localhost:9090 </dev/null 2>&1 | grep -E "(subject=|issuer=)" | head -2

# certmonger tracking
echo ""
echo "7. certmonger tracked certificates:"
sudo getcert list | grep -c "Request ID"
echo "total certificates tracked"

echo ""
echo "=== Check Complete ==="

20.11 Service-Specific Quick References

Cockpit

# Install: dnf install cockpit
# Cert location: /etc/cockpit/ws-certs.d/
# Format: Combined cert+key
# Reload: systemctl restart cockpit.socket
# Test: https://server:9090/

OpenVPN

# Install: dnf install openvpn (EPEL)
# Cert location: /etc/openvpn/server/
# Files: ca.crt, server.crt, server.key
# Start: systemctl start openvpn-server@server
# Test: openvpn --config client.ovpn

strongSwan

# Install: dnf install strongswan
# Cert location: /etc/strongswan/ipsec.d/
# Subdirs: cacerts/, certs/, private/
# Start: systemctl start strongswan
# Test: swanctl --list-sas

HAProxy

# Install: dnf install haproxy
# Cert format: Combined PEM (cert+key+chain)
# Location: /etc/haproxy/certs/
# Config: bind *:443 ssl crt /path/to/bundle.pem
# Test: curl -v https://loadbalancer/

Container Registry

# Run: podman run -d -p 5000:5000 registry:2
# Certs: Mount as volumes (-v)
# Env vars: REGISTRY_HTTP_TLS_CERTIFICATE
#           REGISTRY_HTTP_TLS_KEY
# Test: curl https://registry:5000/v2/_catalog

20.12 Wildcard Certificates for Multiple Services

When to Use Wildcards

Scenario: Multiple subdomains on same server

web.example.com    → Apache
api.example.com    → NGINX
admin.example.com  → Cockpit
mail.example.com   → Postfix

Solution: Use wildcard certificate *.example.com

Generate Wildcard Certificate

#============================================#
# WILDCARD CERTIFICATE
#============================================#

# Generate key
openssl genpkey -algorithm RSA -out wildcard.key -pkeyopt rsa_keygen_bits:2048

# Generate CSR
openssl req -new -key wildcard.key -out wildcard.csr \
  -subj "/CN=*.example.com" \
  -addext "subjectAltName=DNS:*.example.com,DNS:example.com"

# Submit to CA, receive wildcard.crt

# Use for multiple services
sudo cp wildcard.crt /etc/pki/tls/certs/
sudo cp wildcard.key /etc/pki/tls/private/
sudo chmod 600 /etc/pki/tls/private/wildcard.key

# Configure each service to use it
# Apache: SSLCertificateFile /etc/pki/tls/certs/wildcard.crt
# NGINX: ssl_certificate /etc/pki/tls/certs/wildcard.crt
# Postfix: smtpd_tls_cert_file = /etc/pki/tls/certs/wildcard.crt

Pros:

• ✅ One certificate for multiple subdomains
• ✅ Easier management
• ✅ Cost-effective (if purchasing)

Cons:

• ⚠️ If compromised, affects all subdomains
• ⚠️ Doesn’t work for multi-level (..example.com)
• ⚠️ Some security policies prohibit wildcards

20.13 Service Certificate Matrix

Certificate Requirements by Service

20.14 Troubleshooting Quick Guide

Generic Service TLS Troubleshooting

#============================================#
# UNIVERSAL TLS TROUBLESHOOTING
#============================================#

# 1. Identify service and port
ss -tlnp | grep <service>

# 2. Check if TLS is enabled
# (service-specific command)

# 3. Test TLS connection
openssl s_client -connect localhost:<port>
# Or with STARTTLS:
openssl s_client -connect localhost:<port> -starttls <protocol>

# 4. Check certificate files
ls -lZ /path/to/certs/

# 5. Verify ownership/permissions
# - Certificate: 644, owned by service user
# - Key: 600, owned by service user

# 6. Check configuration
# (service-specific config file)

# 7. Check logs
sudo journalctl -u <service> | grep -i tls
sudo tail -f /var/log/<service>/ | grep -i tls

# 8. Test from remote client
openssl s_client -connect server.example.com:<port>

20.15 Centralized Certificate Management

Using certmonger for All Services

#============================================#
# CENTRAL CERTIFICATE MANAGEMENT STRATEGY
#============================================#

# Track all service certificates with certmonger

# Apache
sudo ipa-getcert request -f /etc/pki/tls/certs/apache.crt \
  -k /etc/pki/tls/private/apache.key \
  -K HTTP/$(hostname -f)@REALM \
  -C "systemctl reload httpd"

# NGINX
sudo ipa-getcert request -f /etc/pki/tls/certs/nginx.crt \
  -k /etc/pki/tls/private/nginx.key \
  -K HTTP/$(hostname -f)@REALM \
  -C "systemctl reload nginx"

# Postfix
sudo ipa-getcert request -f /etc/pki/tls/certs/postfix.crt \
  -k /etc/pki/tls/private/postfix.key \
  -K smtp/$(hostname -f)@REALM \
  -C "postfix reload"

# OpenLDAP
sudo ipa-getcert request -f /etc/openldap/certs/ldap.crt \
  -k /etc/openldap/certs/ldap.key \
  -K ldap/$(hostname -f)@REALM \
  -o ldap:ldap \
  -m 600 \
  -C "systemctl restart slapd"

# Monitor all
sudo getcert list

Benefits:

• ✅ Single tool for all services
• ✅ Automatic renewal
• ✅ Centralized monitoring
• ✅ Consistent approach

20.16 Key Takeaways

1. Many services use certificates beyond just web servers
2. Each service has unique requirements - Check ownership, permissions
3. certmonger works with most services for automation
4. Wildcard certificates can simplify multi-service setups
5. Test each service independently
6. Centralized tracking with certmonger recommended
7. Document service-specific configurations

Quick Reference Card

┌───────────────────────────────────────────────────────────────┐
│ OTHER SERVICES CERTIFICATE QUICK REFERENCE                    │
├───────────────────────────────────────────────────────────────┤
│ Cockpit:     /etc/cockpit/ws-certs.d/NN-name.cert             │
│              (combined cert+key)                              │
│                                                               │
│ OpenVPN:     /etc/openvpn/server/{ca,server}.{crt,key}        │
│              Complex PKI with client certs                    │
│                                                               │
│ strongSwan:  /etc/strongswan/ipsec.d/{cacerts,certs,private}/ │
│              IPsec-specific configuration                     │
│                                                               │
│ HAProxy:     Combined PEM (cert+key+chain in one file)        │
│              /etc/haproxy/certs/bundle.pem                    │
│                                                               │
│ Registry:    Environment variables for container              │
│              REGISTRY_HTTP_TLS_CERTIFICATE/KEY                │
│                                                               │
│ Generic:     Check ownership, permissions, SELinux            │
│              Test with: openssl s_client -connect :port       │
└───────────────────────────────────────────────────────────────┘

✅ Use certmonger for automation where possible
✅ Each service has unique file format/location requirements

Chapter Navigation

Chapter 21: Service Certificate Best Practices

Critical for Operations: Learn the best practices that prevent 90% of certificate problems before they happen.

21.1 The Cost of Poor Certificate Management

Real-world impacts:

• ❌ Expired certificate → Website down (revenue loss)
• ❌ Wrong permissions → Service fails to start (downtime)
• ❌ No backup → CA failure means manual reissue (hours/days)
• ❌ Poor naming → Confusion during incident (delayed response)
• ❌ No monitoring → Surprise expiration (emergency response)

This chapter prevents these issues.

21.2 File Organization Best Practices

Standard Directory Structure

/etc/pki/tls/
├── certs/                      # Certificate files (public)
│   ├── service-name.crt        # Actual certificates
│   ├── service-name-chain.crt  # With intermediate chain
│   └── ca-bundle.crt           # CA bundle
│
├── private/                    # Private keys (protected!)
│   └── service-name.key        # Private keys (mode 600)
│
├── csr/                        # Certificate requests (optional)
│   └── service-name.csr        # CSRs for tracking
│
└── backup/                     # Backups (optional but recommended)
    └── YYYY-MM-DD/
        ├── service-name.crt
        └── service-name.key

Naming Conventions

Good naming prevents confusion:

# ✅ GOOD - Clear, descriptive
/etc/pki/tls/certs/web01-example-com.crt
/etc/pki/tls/certs/mail-smtp-example-com.crt
/etc/pki/tls/certs/ldap-primary-example-com.crt

# ❌ BAD - Unclear, generic
/etc/pki/tls/certs/cert1.crt
/etc/pki/tls/certs/new.crt
/etc/pki/tls/certs/temp.crt

Naming pattern:

[service]-[hostname/function]-[domain].crt
[service]-[hostname/function]-[domain].key

Examples:
apache-web01-example-com.crt
nginx-www-example-com.crt
postfix-mail-example-com.crt
ldap-dir01-example-com.crt
postgresql-db-primary-example-com.crt

File Permission Standards

#============================================#
# CRITICAL: Proper Permissions
#============================================#

# Certificates (public) - readable by all
/etc/pki/tls/certs/*.crt           → 644 (rw-r--r--)
/etc/pki/tls/certs/                → 755 (rwxr-xr-x)

# Private keys (secret!) - only readable by owner
/etc/pki/tls/private/*.key         → 600 (rw-------)
/etc/pki/tls/private/              → 711 (rwx--x--x)

# Service-specific keys - owned by service user
/etc/pki/tls/private/apache.key    → 600, owner: root or apache
/etc/pki/tls/private/postgres.key  → 600, owner: postgres

Set permissions script:

#!/bin/bash
# set-cert-permissions.sh
# Sets proper permissions on certificate files

CERT_DIR="/etc/pki/tls/certs"
KEY_DIR="/etc/pki/tls/private"

# Certificate directory
chmod 755 "$CERT_DIR"
chmod 644 "$CERT_DIR"/*.crt 2>/dev/null

# Private key directory
chmod 711 "$KEY_DIR"
chmod 600 "$KEY_DIR"/*.key 2>/dev/null

# Verify
echo "Certificate permissions:"
ls -ld "$CERT_DIR" "$CERT_DIR"/*.crt 2>/dev/null

echo ""
echo "Private key permissions:"
ls -ld "$KEY_DIR" "$KEY_DIR"/*.key 2>/dev/null

# Check for overly permissive keys
echo ""
echo "Checking for security issues:"
find "$KEY_DIR" -type f -not -perm 600 -ls 2>/dev/null && \
  echo "⚠️ WARNING: Some keys have incorrect permissions!" || \
  echo "✅ All keys properly protected"

21.3 Certificate Lifecycle Management

Renewal Timeline

Certificate Lifecycle (365 day validity):

Day   0: Certificate issued
Day  30: First renewal reminder (335 days left)
Day  60: Second reminder (305 days left)
Day 300: Critical renewal window starts (65 days left)
Day 330: URGENT - Renewal needed (35 days left)
Day 350: CRITICAL - Renewal overdue (15 days left)
Day 365: EXPIRED - Service outage!

Recommended Actions:
- Days 300-330: Plan and execute renewal
- Days 330-350: Emergency renewal if missed
- Days 350+: Incident response, temporary cert

Renewal Strategies

Strategy 1: Automated (Recommended)

# Using certmonger (RHEL)
sudo getcert request \
  -f /etc/pki/tls/certs/web01-example-com.crt \
  -k /etc/pki/tls/private/web01-example-com.key \
  -D web.example.com \
  -K host/web.example.com@REALM \
  -C "systemctl reload httpd"  # Auto-reload service

# Auto-renewal happens at 2/3 of cert lifetime
# 365 day cert → renews at day 243 (122 days remaining)

Strategy 2: Scheduled Manual Renewal

# Cron job for manual renewal check
# /etc/cron.weekly/check-certificates

#!/bin/bash
# Check certificates expiring in 60 days
find /etc/pki/tls/certs/ -name "*.crt" | while read cert; do
  if openssl x509 -in "$cert" -noout -checkend $((86400*60)); then
    echo "✅ $cert: OK"
  else
    echo "⚠️ $cert: Expires within 60 days!"
    # Send alert
    mail -s "Certificate Expiring Soon: $cert" admin@example.com
  fi
done

Strategy 3: Calendar Reminders

# For environments without automation
# Create calendar entries:
# - 90 days before expiration: Start renewal
# - 60 days before: Verify renewal in progress
# - 30 days before: Complete renewal
# - 7 days before: Emergency if not done

21.4 Certificate Metadata Tracking

Certificate Inventory

Maintain a certificate inventory (spreadsheet or database):

Service,Hostname,Certificate_Path,Key_Path,Issuer,Issue_Date,Expiry_Date,SANs,Owner,Notes
Apache,web01,/etc/pki/tls/certs/web01-example-com.crt,/etc/pki/tls/private/web01.key,Internal CA,2024-01-01,2025-01-01,"web01.example.com,www.example.com",John Doe,Production
NGINX,web02,/etc/pki/tls/certs/web02-example-com.crt,/etc/pki/tls/private/web02.key,Let's Encrypt,2024-06-15,2024-09-15,"web02.example.com",Jane Smith,Staging

Generate inventory script:

#!/bin/bash
# generate-cert-inventory.sh
# Creates certificate inventory from system

echo "Service,Hostname,Certificate_Path,Issuer,Issue_Date,Expiry_Date,Days_Remaining"

# Scan common certificate locations
for cert in /etc/pki/tls/certs/*.crt /etc/httpd/conf/ssl/*.crt /etc/nginx/ssl/*.crt; do
  [ -f "$cert" ] || continue

  subject=$(openssl x509 -in "$cert" -noout -subject 2>/dev/null | sed 's/subject=//')
  issuer=$(openssl x509 -in "$cert" -noout -issuer 2>/dev/null | sed 's/issuer=//')
  notbefore=$(openssl x509 -in "$cert" -noout -startdate 2>/dev/null | cut -d= -f2)
  notafter=$(openssl x509 -in "$cert" -noout -enddate 2>/dev/null | cut -d= -f2)

  # Calculate days remaining
  expiry_epoch=$(date -d "$notafter" +%s 2>/dev/null)
  now_epoch=$(date +%s)
  days_remaining=$(( ($expiry_epoch - $now_epoch) / 86400 ))

  # Determine service from path
  service="Unknown"
  [[ "$cert" =~ httpd ]] && service="Apache"
  [[ "$cert" =~ nginx ]] && service="NGINX"

  echo "$service,$(hostname),$cert,\"$issuer\",$notbefore,$notafter,$days_remaining"
done

21.5 Backup and Recovery

What to Backup

Critical files to backup:
✅ Private keys (.key files)
✅ Certificates (.crt files)
✅ CA certificates
✅ Certificate chains
✅ CSRs (for reference)
✅ Configuration files (Apache ssl.conf, etc.)
⚠️ NOT passwords or passphrases (store separately in vault)

Backup Script

#!/bin/bash
# backup-certificates.sh
# Backs up all certificates and keys

BACKUP_DIR="/var/backups/certificates"
DATE=$(date +%Y-%m-%d)
BACKUP_PATH="$BACKUP_DIR/$DATE"

# Create backup directory
mkdir -p "$BACKUP_PATH"

# Backup certificates
echo "Backing up certificates..."
cp -a /etc/pki/tls/certs/*.crt "$BACKUP_PATH/" 2>/dev/null

# Backup private keys (encrypted!)
echo "Backing up private keys..."
tar czf - /etc/pki/tls/private/*.key 2>/dev/null | \
  openssl enc -aes-256-cbc -salt -out "$BACKUP_PATH/keys.tar.gz.enc" -pass pass:CHANGEME

# Backup configuration files
echo "Backing up configs..."
cp -a /etc/httpd/conf.d/ssl.conf "$BACKUP_PATH/" 2>/dev/null
cp -a /etc/nginx/nginx.conf "$BACKUP_PATH/" 2>/dev/null

# Create inventory
ls -lh "$BACKUP_PATH"

# Set permissions
chmod 700 "$BACKUP_PATH"

echo "✅ Backup complete: $BACKUP_PATH"
echo "⚠️ Remember to change encryption password!"

Recovery Procedure

#============================================#
# CERTIFICATE RECOVERY PROCEDURE
#============================================#

# 1. Stop affected service
sudo systemctl stop httpd

# 2. Restore certificate
sudo cp /var/backups/certificates/2024-11-15/web.crt /etc/pki/tls/certs/

# 3. Restore private key (decrypt)
cd /var/backups/certificates/2024-11-15/
openssl enc -aes-256-cbc -d -in keys.tar.gz.enc -pass pass:CHANGEME | \
  sudo tar xzf - -C /

# 4. Set permissions
sudo chmod 600 /etc/pki/tls/private/*.key
sudo chmod 644 /etc/pki/tls/certs/*.crt

# 5. Verify files
sudo openssl x509 -in /etc/pki/tls/certs/web-example-com.crt -noout -text
sudo openssl rsa -in /etc/pki/tls/private/web-example-com.key -check

# 6. Start service
sudo systemctl start httpd

# 7. Test
curl -v https://localhost/

21.6 Security Best Practices

Private Key Protection

#============================================#
# PRIVATE KEY SECURITY CHECKLIST
#============================================#

✅ Permissions: 600 (or 400 for extra protection)
✅ Ownership: root or service user only
✅ Location: /etc/pki/tls/private/ (mode 711)
✅ SELinux: Proper context (cert_t)
✅ Backup: Encrypted at rest
✅ Never: Email, paste in tickets, commit to git
✅ Never: Share between systems (generate new)
✅ Audit: Log access with auditd

# Verify security
ls -lZ /etc/pki/tls/private/*.key
# -rw------- root root unconfined_u:object_r:cert_t:s0 server.key

Key Generation Best Practices

#============================================#
# GENERATE SECURE KEYS
#============================================#

# RSA 2048 (minimum for RHEL 8+)
openssl genpkey -algorithm RSA -out server.key -pkeyopt rsa_keygen_bits:2048

# RSA 4096 (recommended for long-lived certs)
openssl genpkey -algorithm RSA -out server.key -pkeyopt rsa_keygen_bits:4096

# EC P-256 (modern, smaller, fast)
openssl genpkey -algorithm EC -out server.key -pkeyopt ec_paramgen_curve:P-256

# Immediately set permissions!
chmod 600 server.key

# ❌ NEVER do this:
# openssl genrsa -out server.key 1024   # Too weak!
# chmod 644 server.key                  # Too permissive!

Certificate Validation Before Deployment

#!/bin/bash
# validate-certificate.sh
# Validates certificate before deployment

CERT=$1
KEY=$2

echo "=== Pre-Deployment Certificate Validation ==="

# Check 1: Certificate file exists and readable
if [ ! -f "$CERT" ]; then
  echo "❌ Certificate file not found: $CERT"
  exit 1
fi

# Check 2: Private key exists and readable
if [ ! -f "$KEY" ]; then
  echo "❌ Private key not found: $KEY"
  exit 1
fi

# Check 3: Certificate is valid X.509
if ! openssl x509 -in "$CERT" -noout 2>/dev/null; then
  echo "❌ Invalid X.509 certificate"
  exit 1
fi

# Check 4: Certificate not expired
if ! openssl x509 -in "$CERT" -noout -checkend 0; then
  echo "❌ Certificate is expired!"
  exit 1
fi

# Check 5: Certificate/key pair match
CERT_MOD=$(openssl x509 -noout -modulus -in "$CERT" | openssl md5)
KEY_MOD=$(openssl rsa -noout -modulus -in "$KEY" 2>/dev/null | openssl md5)

if [ "$CERT_MOD" != "$KEY_MOD" ]; then
  echo "❌ Certificate and key do not match!"
  exit 1
fi

# Check 6: SANs present (required for modern browsers)
if ! openssl x509 -in "$CERT" -noout -ext subjectAltName 2>/dev/null | grep -q "DNS:"; then
  echo "⚠️ WARNING: No Subject Alternative Names found"
fi

# Check 7: Strong signature algorithm
SIG_ALG=$(openssl x509 -in "$CERT" -noout -text | grep "Signature Algorithm" | head -2)
if echo "$SIG_ALG" | grep -qi "sha1\|md5"; then
  echo "❌ Weak signature algorithm: $SIG_ALG"
  exit 1
fi

# Check 8: Adequate key size
KEY_SIZE=$(openssl x509 -in "$CERT" -noout -text | grep "Public-Key:" | grep -oP '\d+')
if [ "$KEY_SIZE" -lt 2048 ]; then
  echo "❌ Key size too small: $KEY_SIZE bits (minimum 2048)"
  exit 1
fi

echo ""
echo "✅ Certificate validation passed!"
echo "   Subject: $(openssl x509 -in "$CERT" -noout -subject)"
echo "   Issuer: $(openssl x509 -in "$CERT" -noout -issuer)"
echo "   Expires: $(openssl x509 -in "$CERT" -noout -enddate | cut -d= -f2)"
echo "   Key Size: $KEY_SIZE bits"

21.7 Multi-Service Coordination

When Multiple Services Share Certificates

# Scenario: Load balancer + multiple web servers

# Problem: Certificate on LB, services behind need same CN/SANs

# Solution 1: Use same certificate on all (if hostnames match)
# web01, web02, web03 all use cert for: web.example.com

# Solution 2: Wildcard certificate
# *.example.com works for web01.example.com, web02.example.com, etc.

# Solution 3: Comprehensive SANs
# Single cert with SANs: web.example.com, web01.example.com, web02.example.com

Certificate Deployment Workflow

#============================================#
# MULTI-SERVER DEPLOYMENT
#============================================#

# Step 1: Generate certificate on management node
openssl genpkey -algorithm RSA -out web.key -pkeyopt rsa_keygen_bits:2048
openssl req -new -key web.key -out web.csr \
  -subj "/CN=web.example.com" \
  -addext "subjectAltName=DNS:web.example.com,DNS:web01.example.com,DNS:web02.example.com"

# Step 2: Get certificate from CA
# (submit web.csr to CA, receive web.crt)

# Step 3: Validate locally
./validate-certificate.sh web.crt web.key

# Step 4: Distribute securely
for host in web01 web02 web03; do
  scp web.crt root@$host:/etc/pki/tls/certs/
  scp web.key root@$host:/etc/pki/tls/private/
  ssh root@$host "chmod 644 /etc/pki/tls/certs/web-example-com.crt"
  ssh root@$host "chmod 600 /etc/pki/tls/private/web-example-com.key"
done

# Step 5: Reload services
for host in web01 web02 web03; do
  ssh root@$host "systemctl reload httpd"
done

# Step 6: Test each server
for host in web01 web02 web03; do
  echo "Testing $host..."
  curl -vk https://$host/ 2>&1 | grep "subject:"
done

21.8 Documentation Standards

Certificate Documentation Template

## Certificate: web.example.com

### Basic Information
- **Service:** Apache (httpd)
- **Server:** web01.example.com
- **Certificate Path:** `/etc/pki/tls/certs/web-example-com.crt`
- **Key Path:** `/etc/pki/tls/private/web-example-com.key`
- **Owner:** Web Team (webadmin@example.com)

### Certificate Details
- **Common Name (CN):** web.example.com
- **SANs:** web.example.com, www.example.com
- **Issuer:** Internal CA (ca.example.com)
- **Issue Date:** 2024-01-01
- **Expiry Date:** 2025-01-01
- **Key Type:** RSA 2048

### Renewal Process
- **Method:** certmonger automatic
- **Renewal Window:** 65 days before expiry
- **Post-Renewal:** `systemctl reload httpd`
- **Contact:** webadmin@example.com

### Service Configuration
- **Config File:** `/etc/httpd/conf.d/ssl.conf`
- **Service:** `httpd.service`
- **Restart Command:** `systemctl reload httpd`

### Troubleshooting
- **Logs:** `/var/log/httpd/ssl_error_log`
- **Test Command:** `curl -v https://web.example.com/`
- **Common Issues:** None reported

### Change History
- 2024-01-01: Initial deployment
- 2024-06-15: Added www.example.com SAN

21.9 Monitoring and Alerting

What to Monitor

✅ Certificate expiration (60, 30, 7 days before)
✅ Certificate validity (not expired, not yet valid)
✅ Certificate/key pair match
✅ Certificate trust chain
✅ Service health (is it using the cert?)
✅ certmonger tracking status
✅ Renewal success/failure

Simple Monitoring Script

#!/bin/bash
# monitor-certificates.sh
# Simple certificate monitoring

WARN_DAYS=30
CRIT_DAYS=7
EMAIL="admin@example.com"

check_cert() {
  local cert=$1
  local name=$(basename "$cert")

  # Check if expires within warning period
  if ! openssl x509 -in "$cert" -noout -checkend $((86400*WARN_DAYS)); then
    if ! openssl x509 -in "$cert" -noout -checkend $((86400*CRIT_DAYS)); then
      echo "🚨 CRITICAL: $name expires within $CRIT_DAYS days!"
      return 2
    else
      echo "⚠️ WARNING: $name expires within $WARN_DAYS days"
      return 1
    fi
  fi

  return 0
}

# Check all certificates
WARNINGS=0
CRITICALS=0

for cert in /etc/pki/tls/certs/*.crt; do
  [ -f "$cert" ] || continue
  check_cert "$cert"
  ret=$?
  [ $ret -eq 1 ] && ((WARNINGS++))
  [ $ret -eq 2 ] && ((CRITICALS++))
done

# Alert if issues found
if [ $CRITICALS -gt 0 ] || [ $WARNINGS -gt 0 ]; then
  echo "Certificate issues found: $CRITICALS critical, $WARNINGS warnings" | \
    mail -s "Certificate Alert: $(hostname)" "$EMAIL"
fi

21.10 Incident Response Procedures

Certificate Expiration Incident

#============================================#
# EXPIRED CERTIFICATE EMERGENCY
#============================================#

# Step 1: Assess impact
systemctl status httpd
journalctl -xe | grep -i cert

# Step 2: Quick fix - Get temporary cert
# Option A: Self-signed (for internal only!)
openssl req -x509 -nodes -days 30 -newkey rsa:2048 \
  -keyout /etc/pki/tls/private/temp-web01-example-com.key \
  -out /etc/pki/tls/certs/temp-web01-example-com.crt \
  -subj "/CN=$(hostname)"

# Option B: Restore from backup
cp /var/backups/certificates/latest/*.crt /etc/pki/tls/certs/
cp /var/backups/certificates/latest/*.key /etc/pki/tls/private/

# Step 3: Update service config to use temp cert
# Edit /etc/httpd/conf.d/ssl.conf
# SSLCertificateFile /etc/pki/tls/certs/temp-web01-example-com.crt
# SSLCertificateKeyFile /etc/pki/tls/private/temp-web01-example-com.key

# Step 4: Restart service
systemctl restart httpd

# Step 5: Get proper certificate ASAP
# Follow normal cert request process

# Step 6: Document incident
# What happened, why, how fixed, prevention

21.11 Best Practices Checklist

## Certificate Management Checklist

### File Organization
- [ ] Standard directory structure used
- [ ] Consistent naming convention
- [ ] Proper file permissions (600 for keys, 644 for certs)
- [ ] SELinux contexts correct

### Lifecycle Management
- [ ] Renewal process defined and documented
- [ ] Renewal reminders set (60, 30, 7 days)
- [ ] Automated renewal if possible (certmonger)
- [ ] Post-renewal actions defined

### Security
- [ ] Private keys protected (600 permissions)
- [ ] Keys never shared/emailed
- [ ] Strong key algorithm (RSA 2048+ or EC P-256)
- [ ] Strong signature (SHA-256+)

### Backup
- [ ] Certificates backed up
- [ ] Private keys backed up (encrypted)
- [ ] Backup tested and validated
- [ ] Restore procedure documented

### Documentation
- [ ] Certificate inventory maintained
- [ ] Each certificate documented
- [ ] Procedures written
- [ ] Contacts listed

### Monitoring
- [ ] Expiration monitoring enabled
- [ ] Alerts configured
- [ ] Health checks in place
- [ ] Incident response plan ready

### Validation
- [ ] Pre-deployment validation
- [ ] Post-deployment testing
- [ ] Regular audits scheduled

21.12 Key Takeaways

1. Organization prevents confusion - Consistent structure and naming
2. Permissions are critical - 600 for keys, 644 for certs
3. Automate renewal - Use certmonger whenever possible
4. Backup everything - But encrypt private keys
5. Document thoroughly - Your future self will thank you
6. Monitor proactively - Don’t wait for expiration
7. Validate before deploy - Catch issues early
8. Plan for incidents - Have recovery procedures ready

Quick Reference

┌─────────────────────────────────────────────────────────────┐
│ SERVICE CERTIFICATE BEST PRACTICES                          │
├─────────────────────────────────────────────────────────────┤
│ Files:     /etc/pki/tls/certs/*.crt (644)                   │
│            /etc/pki/tls/private/*.key (600)                 │
│ Naming:    [service]-[host]-[domain].[crt|key]              │
│ Renewal:   Automate with certmonger                         │
│ Backup:    Daily, encrypted, tested                         │
│ Monitor:   60, 30, 7 days before expiry                     │
│ Validate:  Before every deployment                          │
│ Document:  Everything, everyone                             │
└─────────────────────────────────────────────────────────────┘

Chapter Navigation

Chapter 22: certmonger Mastery

Set It and Forget It: certmonger is RHEL’s built-in certificate automation tool. Master it and you’ll never manually renew a certificate again.

22.1 What is certmonger?

certmonger is a certificate tracking and automatic renewal daemon for RHEL.

Think of it as:

• 📋 Certificate tracker - Monitors expiration dates
• 🔄 Auto-renewer - Renews before expiry
• 🔗 CA integrator - Works with FreeIPA, local/internal CAs, and custom external CA helpers
• ⚙️ Service integration - Runs commands after renewal

Why certmonger?

Without certmonger:

❌ Manual tracking of expiration dates
❌ Calendar reminders to renew
❌ Manual CSR generation
❌ Manual service restart after renewal
❌ Risk of missing renewals → outages

With certmonger:

✅ Automatic tracking
✅ Automatic renewal
✅ Automatic service reload
✅ Centralized monitoring
✅ No manual intervention!

22.2 Installation and Setup

Installation by RHEL Version

#============================================#
# INSTALL CERTMONGER
#============================================#

# RHEL 8/9/10
sudo dnf install certmonger -y

# RHEL 7
# sudo yum install certmonger -y

# Enable and start
sudo systemctl enable certmonger
sudo systemctl start certmonger

# Verify
systemctl status certmonger
sudo getcert list  # Should show empty list initially

22.3 Basic Usage

Requesting a Certificate

#============================================#
# BASIC CERTIFICATE REQUEST
#============================================#

# Self-signed (for testing)
sudo getcert request \
  -f /etc/pki/tls/certs/test.crt \
  -k /etc/pki/tls/private/test.key

# From FreeIPA / IdM
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/web.crt \
  -k /etc/pki/tls/private/web.key \
  -K HTTP/$(hostname -f)@REALM \
  -D $(hostname -f)

# For public Let's Encrypt certificates, use certbot (Chapter 24).
# certmonger is the native choice for IPA, local CA, and helper-based workflows.

Checking Status

#============================================#
# CHECK CERTIFICATE STATUS
#============================================#

# List all tracked certificates
sudo getcert list

# Check specific certificate by file
sudo getcert list -f /etc/pki/tls/certs/web.crt

# Check by request ID
sudo getcert list -i 20240101000000

# Verbose output
sudo getcert list -v

Status Values:

• MONITORING: ✅ Certificate issued, tracking expiration
• SUBMITTING: 🔄 Submitting request to CA
• CA_UNREACHABLE: ❌ Can’t reach CA server
• CA_REJECTED: ❌ CA rejected request
• NEED_KEY_GEN_PIN: ⏸️ Waiting for PIN (HSM/token)
• PRE_SAVE_COMMAND: 🔄 Running pre-save script
• POST_SAVE_COMMAND: 🔄 Running post-save script

22.4 Advanced Options

Complete Request with All Options

#============================================#
# COMPLETE CERTMONGER REQUEST
#============================================#

sudo ipa-getcert request \
  -f /etc/pki/tls/certs/web.example.com.crt \              # Certificate file
  -k /etc/pki/tls/private/web.example.com.key \            # Private key file
  -K HTTP/web.example.com@EXAMPLE.COM \                    # Kerberos principal
  -D web.example.com \                                     # DNS name (SAN)
  -D www.example.com \                                     # Additional SAN
  -D api.example.com \                                     # Another SAN
  -U id-kp-serverAuth \                                    # Extended key usage
  -N CN=web.example.com,O=Example,C=US \                   # Subject DN
  -g 2048 \                                                # Key size
  -G rsa \                                                 # Key type
  -T caIPAserviceCert \                                    # IPA profile
  -C "systemctl reload httpd" \                            # Post-save command
  -B "systemctl stop httpd" \                              # Pre-save command
  -v \                                                     # Verbose
  -w                                                       # Wait for completion

# Check status
sudo getcert list -f /etc/pki/tls/certs/web.example.com.crt

Key Options Explained:

22.5 Working with Different CAs

FreeIPA (Recommended for Internal)

#============================================#
# CERTMONGER + FREEIPA
#============================================#

# Prerequisites: System enrolled to IPA
ipa-client-install

# Request certificate
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/internal.crt \
  -k /etc/pki/tls/private/internal.key \
  -K HTTP/$(hostname -f)@REALM \
  -D $(hostname -f) \
  -C "systemctl reload httpd"

# certmonger automatically:
# ✅ Submits request to IPA CA
# ✅ Obtains certificate
# ✅ Saves to file
# ✅ Runs reload command
# ✅ Tracks expiration
# ✅ Renews ~28 days before expiry

Public ACME Requires certbot

#============================================#
# PUBLIC ACME VS NATIVE CERTMONGER WORKFLOWS
#============================================#

# Public Let's Encrypt certificate:
# Use certbot, not a fake certmonger CA profile.
sudo certbot certonly --apache -d public.example.com

# Native FreeIPA / IdM certificate:
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/internal.crt \
  -k /etc/pki/tls/private/internal.key \
  -K HTTP/$(hostname -f)@REALM \
  -D $(hostname -f) \
  -C "systemctl reload httpd"

External CA (Manual Submission)

#============================================#
# CERTMONGER WITH EXTERNAL CA
#============================================#

# Configure external CA helper
sudo getcert add-ca -c external-ca \
  -e '/usr/local/bin/external-ca-submit.sh'

# Request certificate
sudo getcert request \
  -c external-ca \
  -f /etc/pki/tls/certs/external.crt \
  -k /etc/pki/tls/private/external.key

# Helper script must:
# 1. Read CSR from stdin
# 2. Submit to external CA
# 3. Return certificate on stdout

22.6 Managing Tracked Certificates

Modify Tracking

#============================================#
# MODIFY EXISTING CERTIFICATE TRACKING
#============================================#

# Update post-save command without rekeying
sudo getcert stop-tracking -f /etc/pki/tls/certs/web.crt
sudo getcert start-tracking \
  -f /etc/pki/tls/certs/web.crt \
  -k /etc/pki/tls/private/web.key \
  -C "systemctl reload httpd"
# Re-add -c, -K, -D, etc. if your original tracking entry used them

# Add additional SAN
sudo getcert resubmit -f /etc/pki/tls/certs/web.crt \
  -D additional.example.com

# Stop tracking (keep certificate)
sudo getcert stop-tracking -f /etc/pki/tls/certs/web.crt

# Remove completely
sudo getcert stop-tracking -f /etc/pki/tls/certs/web.crt -r

# Start tracking existing certificate
sudo getcert start-tracking \
  -f /etc/pki/tls/certs/existing.crt \
  -k /etc/pki/tls/private/existing.key

Force Renewal

#============================================#
# FORCE IMMEDIATE RENEWAL
#============================================#

# By file path
sudo ipa-getcert resubmit -f /etc/pki/tls/certs/web.crt

# By request ID
sudo getcert resubmit -i 20240101000000

# Wait for renewal
sudo getcert list -f /etc/pki/tls/certs/web.crt
# Wait for status: MONITORING

22.7 Renewal Timing

Understanding Renewal Windows

Certificate Lifecycle (365 days):

Day   0: Certificate issued
      │
      │ [Normal operation]
      │
Day 243: Renewal window starts (certmonger attempts renewal)
      │ (2/3 of cert lifetime: 365 × 2/3 ≈ 243 days)
      │
      │ [Renewal attempts every 8 hours if CA available]
      │
Day 335: Warning if not yet renewed (30 days left)
      │
Day 350: Critical if not yet renewed (15 days left)
      │
Day 365: Certificate expires → SERVICE OUTAGE if not renewed!

Default Behavior:

• Renewal starts at 2/3 of certificate lifetime
• 365-day cert → Renews at day 243 (122 days remaining)
• 90-day cert → Renews at day 60 (30 days remaining)

22.8 Post-Save Commands

Reload vs Restart

#============================================#
# POST-SAVE COMMAND STRATEGIES
#============================================#

# PREFER: reload (no downtime)
-C "systemctl reload httpd"
-C "systemctl reload nginx"
-C "postfix reload"

# SOMETIMES NEEDED: restart
-C "systemctl restart slapd"  # OpenLDAP requires restart
-C "systemctl restart postgresql"  # PostgreSQL requires restart

# MULTIPLE COMMANDS: Use script
-C "/usr/local/bin/after-cert-renewal.sh"

# Script example:
#!/bin/bash
# /usr/local/bin/after-cert-renewal.sh
systemctl reload httpd
systemctl reload nginx
systemctl reload postfix
logger "Certificates renewed via certmonger"

22.9 Troubleshooting certmonger

Common Issues

Issue 1: CA_UNREACHABLE

# Symptom
sudo getcert list
# status: CA_UNREACHABLE

# Diagnosis
# For FreeIPA:
ipa ping  # Check IPA connectivity
klist  # Check Kerberos ticket

# Fix
kinit -k host/$(hostname -f)@REALM  # Renew host ticket
sudo ipa-getcert resubmit -f /etc/pki/tls/certs/web.crt

# Check IPA server
ssh ipa-server "sudo ipactl status"

Issue 2: CA_REJECTED

# Symptom
sudo getcert list
# status: CA_REJECTED
# ca-error: Server at https://ipa.example.com/ipa/xml unwilling to issue certificate

# Common causes:
# 1. Service principal doesn't exist
ipa service-show HTTP/$(hostname -f)
# If not found:
ipa service-add HTTP/$(hostname -f)

# 2. Host not enrolled
ipa host-show $(hostname -f)

# 3. Permission issue
# Check IPA permissions

# Retry
sudo ipa-getcert resubmit -f /etc/pki/tls/certs/web.crt

Issue 3: Renewal Not Happening

# Check certmonger is running
systemctl status certmonger

# Check certificate status
sudo getcert list -f /etc/pki/tls/certs/web.crt

# Check certmonger logs
sudo journalctl -u certmonger -f

# Force renewal
sudo ipa-getcert resubmit -f /etc/pki/tls/certs/web.crt

# Check renewal window
# Certificate renews at 2/3 of lifetime
# Check "expires" date in getcert list output

22.10 IdM ACME vs Public Let’s Encrypt

Keep the Workflows Separate

#============================================#
# CHOOSE THE RIGHT TOOL FOR THE RIGHT CA
#============================================#

# Public Let's Encrypt certificate:
# Use certbot (see Chapter 24).
sudo certbot certonly --apache -d public.example.com -d www.public.example.com

# Native FreeIPA / IdM certificate:
# Use certmonger + ipa-getcert.
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/internal.example.com.crt \
  -k /etc/pki/tls/private/internal.example.com.key \
  -K HTTP/internal.example.com@REALM \
  -D internal.example.com \
  -C "systemctl reload httpd"

# If IdM ACME is enabled, its ACME directory is your IPA server,
# not Let's Encrypt:
sudo certbot certonly \
  --server https://ipa.example.com/acme/directory \
  -d host.example.com

Important distinction:

• Let’s Encrypt = public internet ACME CA
• IdM/FreeIPA ACME = your internal IPA CA exposing an ACME endpoint
• certmonger = native RHEL tracker/renewer for IPA and helper-based workflows

22.11 Monitoring certmonger

Status Monitoring

#============================================#
# MONITOR CERTMONGER
#============================================#

# Overview of all certificates
sudo getcert list

# Count certificates by status
sudo getcert list | grep "status:" | sort | uniq -c

# Find certificates expiring soon (30 days)
for cert in $(sudo getcert list | grep "certificate:" | sed -n "s/.*location='\\([^']*\\)'.*/\\1/p"); do
  if ! openssl x509 -in "$cert" -noout -checkend $((86400*30)) 2>/dev/null; then
    echo "⚠️ Expires soon: $cert"
  fi
done

# Check certmonger logs
sudo journalctl -u certmonger --since today

# Watch for renewal activity
sudo journalctl -u certmonger -f

# Check next renewal time
sudo getcert list | grep -A15 "Request ID" | grep "expires"

Health Check Script

#!/bin/bash
# certmonger-health-check.sh

echo "=== certmonger Health Check ==="

# certmonger running?
if systemctl is-active --quiet certmonger; then
  echo "✅ certmonger is running"
else
  echo "❌ certmonger is NOT running!"
  exit 1
fi

# Count tracked certificates
TOTAL=$(sudo getcert list | grep -c "Request ID")
echo "📋 Tracking $TOTAL certificates"

# Check status breakdown
echo ""
echo "Status breakdown:"
sudo getcert list | grep "status:" | sort | uniq -c

# Check for problems
PROBLEMS=$(sudo getcert list | grep "status:" | grep -v "MONITORING" | wc -l)
if [ $PROBLEMS -gt 0 ]; then
  echo ""
  echo "⚠️ $PROBLEMS certificates need attention:"
  sudo getcert list | grep -B5 "status:" | grep -E "(Request ID|status:)" | grep -v "MONITORING"
fi

# Check expiration warnings
echo ""
echo "Certificates expiring in 30 days:"
sudo getcert list | grep -A10 "Request ID" | grep "expires:" | \
  while read line; do
    # Parse and check expiry
    # (simplified - production script would parse dates properly)
    echo "$line"
  done

22.12 certmonger Configuration

Main Configuration File

#============================================#
# CERTMONGER CONFIGURATION
#============================================#

# Config location
/etc/certmonger/certmonger.conf

# Database location (tracked certificates)
/var/lib/certmonger/

# List configured CAs
sudo getcert list-cas

# Add custom CA
sudo getcert add-ca -c my-ca \
  -e '/usr/local/bin/my-ca-submit.sh'

# Remove CA
sudo getcert remove-ca -c my-ca

22.13 Best Practices

certmonger Best Practices

✅ **Always use post-save commands** (-C flag) to reload services
✅ **Track all production certificates** with certmonger
✅ **Monitor status weekly** with `getcert list`
✅ **Test renewal** before expiry with `resubmit`
✅ **Use verbose mode** (-v) when troubleshooting
✅ **Set up monitoring** for CA_UNREACHABLE status
✅ **Document request IDs** in your certificate inventory
✅ **Use IPA/certmonger for internal** certs, certbot for public Let's Encrypt
✅ **Keep certmonger logs** for audit trail
✅ **Test post-save commands** independently before use

What to Track with certmonger

# ✅ TRACK with certmonger:
- Web server certificates (Apache, NGINX)
- Mail server certificates (Postfix, Dovecot)
- LDAP server certificates (OpenLDAP)
- Application certificates (APIs, microservices)
- Service certificates (any TLS-enabled service)

# ❌ DON'T track with certmonger:
- CA root certificates (managed separately)
- Client certificates for users (different lifecycle)
- Test/temporary certificates
- Certificates you manage with other tools (certbot)

22.14 certmonger vs certbot

When to Use Which?

Recommendation:

• Internal/Enterprise: Use certmonger + FreeIPA
• Public/Simple: Use certbot (but know it requires EPEL)
• Public certificates on any RHEL version: Use certbot for Let’s Encrypt

22.15 Advanced Scenarios

Scenario 1: High Availability Certificate Renewal

# Multiple servers with same service

# Server 1:
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/shared-service.crt \
  -k /etc/pki/tls/private/shared-service.key \
  -K HTTP/service.example.com@REALM \
  -D service.example.com \
  -C "systemctl reload httpd"

# Server 2: Same configuration
# Result: Each server manages its own cert independently
# Or: Use shared cert (copy files, not recommended)

Scenario 2: Wildcard Certificate

# Request wildcard from FreeIPA
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/wildcard.crt \
  -k /etc/pki/tls/private/wildcard.key \
  -K HTTP/*.example.com@REALM \
  -D *.example.com \
  -D example.com \
  -C "/usr/local/bin/reload-all-services.sh"

Scenario 3: EC Keys (Elliptic Curve)

# Request with EC key
sudo ipa-getcert request \
  -f /etc/pki/tls/certs/ec-cert.crt \
  -k /etc/pki/tls/private/ec-cert.key \
  -K HTTP/$(hostname -f)@REALM \
  -G ec \
  -g nistp256  # or nistp384, nistp521

22.16 Key Takeaways

1. certmonger is RHEL’s certificate automation
2. Set it and forget it - Automatic renewal
3. Works best with FreeIPA, internal CAs, and helper-based renewals
4. Post-save commands reload services automatically
5. Tracks expiration and renews at 2/3 of lifetime
6. MONITORING status means all is well
7. getcert list is your main monitoring tool

Quick Reference Card

┌──────────────────────────────────────────────────────────────┐
│ CERTMONGER MASTERY QUICK REFERENCE                           │
├──────────────────────────────────────────────────────────────┤
│ Install:      dnf install certmonger (yum on RHEL 7)         │
│ Start:        systemctl enable --now certmonger              │
│                                                              │
│ Request:      ipa-getcert request -f cert -k key -K principal│
│ List:         getcert list                                   │
│ Status:       getcert list -f /path/to/cert.crt              │
│ Resubmit:     ipa-getcert resubmit -f /path/to/cert.crt      │
│ Stop track:   getcert stop-tracking -f /path/to/cert.crt     │
│                                                              │
│ Status:       MONITORING = ✅ Good                           │
│               CA_UNREACHABLE = ❌ Check IPA/CA               │
│               CA_REJECTED = ❌ Check principal/permissions   │
│                                                              │
│ Logs:         journalctl -u certmonger -f                    │
│ Renewal:      Automatic at 2/3 of cert lifetime              │
│ Post-save:    -C "systemctl reload <service>"                │
└──────────────────────────────────────────────────────────────┘

✅ Native RHEL tool (Red Hat supported)
✅ Perfect for FreeIPA integration
✅ Best fit for FreeIPA and tracked renewal workflows

🧪 Hands-On Lab

Lab 11: certmonger Basics

Automate certificate renewal with certmonger

• 📁 Location: labs/en_US/11-certmonger-basics/
• ⏱️ Time: 30-35 minutes
• 🎯 Level: Intermediate

Chapter Navigation

Chapter 23: Crypto-Policies Deep Dive

Revolutionary Feature: RHEL 8+ crypto-policies provide system-wide cryptographic configuration. Master this and you control security across all applications with one command.

23.1 The Problem crypto-policies Solves

Before Crypto-Policies (RHEL 7)

Configure security individually for EVERY application:

Apache:      /etc/httpd/conf.d/ssl.conf
             SSLProtocol, SSLCipherSuite

NGINX:       /etc/nginx/nginx.conf
             ssl_protocols, ssl_ciphers

Postfix:     /etc/postfix/main.cf
             smtpd_tls_protocols, smtpd_tls_mandatory_ciphers

OpenLDAP:    olcTLSProtocolMin, olcTLSCipherSuite

PostgreSQL:  ssl_min_protocol_version

OpenSSH:     /etc/ssh/sshd_config
             Ciphers, MACs, KexAlgorithms

... and 20+ more applications!

Result: Inconsistent security, configuration nightmare

After Crypto-Policies (RHEL 8/9/10)

✅ Set ONE system-wide policy
✅ All applications automatically comply
✅ Consistent security across the system
✅ Change policy in seconds, not hours

Game changer for enterprise management!

23.2 How Crypto-Policies Work

Architecture

┌──────────────────────────────────────────────────┐
│       update-crypto-policies --set DEFAULT       │
│              (Administrator command)             │
└───────────────────────┬──────────────────────────┘
                        │
                        ▼
┌──────────────────────────────────────────────────┐
│  /etc/crypto-policies/back-ends/                 │
│  (Generated config files for each library)       │
│  ├─ opensslcnf.config                            │
│  ├─ gnutls.config                                │
│  ├─ nss.config                                   │
│  ├─ bind.config                                  │
│  └─ ... more ...                                 │
└───────────────────────┬──────────────────────────┘
                        │
            ┌───────────┼───────────┐
            ▼           ▼           ▼
         OpenSSL     GnuTLS        NSS
            ↓           ↓           ↓
         Apache      NGINX       Firefox
         Postfix     vsftpd      Thunderbird
         OpenSSH     wget        Java apps

Key Insight: Applications read from back-end files, not directly from policy!

23.3 The Four Main Policies

Policy Comparison

DEFAULT Policy Details

# Balanced security and compatibility
Protocols:
  - TLS 1.2
  - TLS 1.3

Minimum Key Sizes:
  - RSA: 2048 bits
  - DH: 2048 bits
  - ECC: secp256r1 (P-256)

Allowed Ciphers:
  - AES-128-GCM
  - AES-256-GCM
  - ChaCha20-Poly1305
  - AES-128-CBC
  - AES-256-CBC

Signature Algorithms:
  - SHA-256
  - SHA-384
  - SHA-512

Blocked:
  - TLS 1.0, 1.1
  - MD5
  - SHA-1 signatures
  - 3DES, RC4, DES
  - RSA < 2048 bits
  - Export ciphers

23.4 Viewing and Changing Policies

Basic Commands

#============================================#
# CRYPTO-POLICY BASIC OPERATIONS
#============================================#

# View current policy
update-crypto-policies --show
# Output: DEFAULT

# List available policies
ls /usr/share/crypto-policies/policies/
# DEFAULT.pol  FUTURE.pol  LEGACY.pol  FIPS.pol

# Set policy
sudo update-crypto-policies --set FUTURE

# You MUST restart services for changes to take effect!
sudo systemctl restart httpd nginx postfix slapd

# Or reboot (ensures everything picks up changes)
sudo reboot

What Happens When You Change Policy

#============================================#
# BEHIND THE SCENES
#============================================#

# Before
update-crypto-policies --show
# DEFAULT

# Change
sudo update-crypto-policies --set FUTURE

# Updates generated:
ls -l /etc/crypto-policies/back-ends/
# -rw-r--r--. opensslcnf.config    ← Updated!
# -rw-r--r--. gnutls.config        ← Updated!
# -rw-r--r--. nss.config           ← Updated!
# -rw-r--r--. bind.config          ← Updated!
# ... all back-ends updated ...

# View OpenSSL config
cat /etc/crypto-policies/back-ends/opensslcnf.config
# Shows actual OpenSSL configuration applied

23.5 Subpolicies (RHEL 9+)

Policy Modifiers

RHEL 9 introduced subpolicies - fine-tune existing policies!

#============================================#
# CRYPTO-POLICY SUBPOLICIES (RHEL 9+)
#============================================#

# Base policy with modifier
sudo update-crypto-policies --set DEFAULT:NO-SHA1

# Multiple modifiers
sudo update-crypto-policies --set DEFAULT:NO-SHA1:GOST

# Available subpolicy modules
ls /usr/share/crypto-policies/policies/modules/
# AD-SUPPORT.pmod
# GOST.pmod
# NO-CAMELLIA.pmod
# NO-SHA1.pmod
# NO-ENFORCE-EMS.pmod
# ...and more

# View module details
cat /usr/share/crypto-policies/policies/modules/NO-SHA1.pmod

Common Subpolicies:

23.6 Creating Custom Policy Modules

Custom Module Example

#============================================#
# CREATE CUSTOM POLICY MODULE
#============================================#

# Create custom module
sudo vi /etc/crypto-policies/policies/modules/CUSTOM-SECURITY.pmod

# Example content:
min_rsa_size = 4096
min_dh_size = 3072
min_dsa_size = 3072
sha1_in_certs = 0
arbitrary_dh_groups = 0
ssh_certs = 0

# Apply
sudo update-crypto-policies --set DEFAULT:CUSTOM-SECURITY

# Restart services
sudo systemctl restart httpd nginx postfix

# Verify
openssl ciphers -v | grep -E "RSA|DH"

23.7 Per-Application Overrides

When to Override

Sometimes ONE application needs different settings than system policy:

Example: Legacy application needs TLS 1.1, but system uses DEFAULT

Apache Override

#============================================#
# APACHE CRYPTO-POLICY OVERRIDE
#============================================#

# /etc/httpd/conf.d/ssl.conf

# Option 1: Include crypto-policy, then override
Include /etc/crypto-policies/back-ends/httpd.config

# Then add overrides:
SSLProtocol all -SSLv3  # Re-enable TLS 1.0/1.1

# Option 2: Completely opt-out
# Don't include crypto-policy file
# Manually configure everything:
SSLProtocol TLSv1.1 TLSv1.2 TLSv1.3
SSLCipherSuite HIGH:!aNULL:!MD5

# ⚠️ Warning: You now manage Apache TLS manually
# System crypto-policy changes won't affect Apache

Better Approach: Create custom policy module instead of per-app overrides!

23.8 Testing Policy Impact

Before Changing Policy

#============================================#
# TEST POLICY CHANGE IMPACT
#============================================#

# 1. Document current state
update-crypto-policies --show > /tmp/current-policy.txt
systemctl list-units --type=service --state=running > /tmp/running-services.txt

# 2. Test applications
curl https://localhost/
psql -h localhost  # etc.

# 3. Change policy on test system first
sudo update-crypto-policies --set FUTURE

# 4. Restart services
sudo systemctl restart httpd nginx postfix

# 5. Test thoroughly
./test-all-services.sh

# 6. If problems: Revert
sudo update-crypto-policies --set DEFAULT

# 7. If successful: Document and deploy to production

23.9 Policy Impact on Certificates

What Policies Control

Crypto-policies affect:

• ✅ TLS protocol versions allowed
• ✅ Cipher suites available
• ✅ Minimum key sizes accepted
• ✅ Signature algorithms allowed
• ✅ Diffie-Hellman parameters
• ✅ Certificate validation strictness

Crypto-policies DON’T affect:

• ❌ Which certificates to use (still configured per service)
• ❌ Certificate file locations
• ❌ CA trust store (that’s update-ca-trust)
• ❌ Certificate issuance

Certificate Compatibility Matrix

23.10 Troubleshooting Crypto-Policies

Common Issues

Issue 1: Application Fails After Policy Change

# Symptom
sudo update-crypto-policies --set FUTURE
sudo systemctl restart httpd
# httpd fails to start

# Diagnosis
sudo journalctl -xe -u httpd | grep -i cipher

# Common cause: Application has hard-coded weak ciphers

# Solution 1: Revert policy
sudo update-crypto-policies --set DEFAULT

# Solution 2: Update application config
# Remove hard-coded cipher specifications

# Solution 3: Create custom policy module

Issue 2: “No Shared Cipher”

# Symptom: Clients can't connect

# Test
openssl s_client -connect server:443

# If shows "no shared cipher":

# Check policy
update-crypto-policies --show

# Test client capabilities
openssl s_client -connect server:443 -cipher 'ALL' -tls1_2

# Temporary fix (not recommended long-term):
sudo update-crypto-policies --set LEGACY

# Proper fix: Update client to support TLS 1.2+ and modern ciphers

Issue 3: Policy Doesn’t Seem to Apply

# Check if application is overriding policy

# Apache
grep -r "SSLProtocol\|SSLCipherSuite" /etc/httpd/
# If found: App is overriding policy

# NGINX
grep -r "ssl_protocols\|ssl_ciphers" /etc/nginx/
# If found: App is overriding policy

# Solution: Remove overrides, let crypto-policy handle it
# Or: Document why override is necessary

23.11 Best Practices

Recommendations

✅ **Use DEFAULT policy** for most environments
✅ **Test before deploying** new policies
✅ **Document policy choices** and reasons
✅ **Restart services** after policy changes
✅ **Avoid per-app overrides** when possible
✅ **Use subpolicies** (RHEL 9+) for fine-tuning
✅ **Monitor for compatibility** issues
✅ **Keep LEGACY temporary** if used
✅ **Plan migrations** when changing policies
✅ **Update clients** rather than weaken policy

When to Use Each Policy

DEFAULT:

• ✅ Most production environments
• ✅ Balanced security/compatibility
• ✅ Recommended starting point
• ✅ Tested and maintained by Red Hat

LEGACY:

• ⚠️ Temporary during migrations only!
• ⚠️ Supporting very old clients
• ⚠️ Testing compatibility issues
• ❌ Never long-term!

FUTURE:

• ✅ High-security environments
• ✅ All clients are modern
• ✅ Want strongest settings
• ✅ Planning for future standards

FIPS:

• ✅ Federal compliance required
• ✅ Government contracts
• ✅ Regulated industries
• ✅ Certification requirements

23.12 FIPS Policy Deep Dive

Enabling FIPS Mode

#============================================#
# ENABLE FIPS MODE
#============================================#

# Check current status
fips-mode-setup --check
# FIPS mode is disabled.

# Enable FIPS mode
sudo fips-mode-setup --enable

# MUST reboot
sudo reboot

# Verify after reboot
fips-mode-setup --check
# FIPS mode is enabled.

# Crypto-policy automatically set to FIPS
update-crypto-policies --show
# FIPS

FIPS Mode Requirements:

• Must be enabled at install OR with fips-mode-setup
• Requires reboot
• Affects entire system
• Only FIPS-approved algorithms available
• Performance impact (~10-20% slower)

FIPS Policy Specifics

# What FIPS policy allows:
✅ TLS 1.2, 1.3
✅ RSA 2048+ bits
✅ AES-128, AES-256 (GCM mode)
✅ SHA-256, SHA-384, SHA-512
✅ ECDHE key exchange

# What FIPS blocks:
❌ TLS 1.0, 1.1
❌ RSA < 2048 bits
❌ 3DES, RC4, DES
❌ MD5, SHA-1
❌ Non-approved algorithms
❌ CBC mode ciphers (in some cases)

23.13 Monitoring and Auditing

Check Policy Compliance

#============================================#
# VERIFY CRYPTO-POLICY COMPLIANCE
#============================================#

# Current policy
update-crypto-policies --show

# Which applications use crypto-policies?
ls -l /etc/crypto-policies/back-ends/

# Verify OpenSSL follows policy
openssl ciphers -v | head -20

# Check specific application config
# Apache
cat /etc/crypto-policies/back-ends/httpd.config

# Test actual connection
openssl s_client -connect localhost:443 -tls1_3

# Verify no overrides
grep -r "SSLProtocol\|SSLCipherSuite" /etc/httpd/ | grep -v crypto-policies
# Should be empty or commented out

23.14 Troubleshooting Workflow

Systematic Approach

Application fails after policy change?
    │
    ├─ Step 1: Identify error
    │   └─ Check logs: journalctl -xe
    │
    ├─ Step 2: Verify policy active
    │   └─ update-crypto-policies --show
    │
    ├─ Step 3: Test with LEGACY
    │   └─ sudo update-crypto-policies --set LEGACY
    │   └─ If works → cipher/protocol issue
    │
    ├─ Step 4: Identify incompatibility
    │   └─ openssl s_client -cipher 'ALL' -tls1
    │   └─ Find what client/server needs
    │
    ├─ Step 5: Choose solution
    │   ├─ A) Update client (best)
    │   ├─ B) Create custom module (good)
    │   └─ C) Per-app override (last resort)
    │
    └─ Step 6: Document and deploy
        └─ Why override needed, plan to remove

23.15 Key Takeaways

1. Crypto-policies are RHEL 8+ only (not in RHEL 7)
2. DEFAULT policy is recommended for most cases
3. Changes require service restarts to take effect
4. Affects ALL crypto applications system-wide
5. Subpolicies provide fine-tuning (RHEL 9+)
6. Avoid per-app overrides when possible
7. Test before deploying new policies
8. LEGACY is temporary only!

Quick Reference Card

┌──────────────────────────────────────────────────────────────┐
│ CRYPTO-POLICIES QUICK REFERENCE                              │
├──────────────────────────────────────────────────────────────┤
│ Available:    RHEL 8, 9, 10 only (not RHEL 7)                │
│                                                              │
│ View:         update-crypto-policies --show                  │
│ Set:          sudo update-crypto-policies --set <POLICY>     │
│ Policies:     DEFAULT, LEGACY, FUTURE, FIPS                  │
│                                                              │
│ Subpolicy:    update-crypto-policies --set DEFAULT:NO-SHA1   │
│               (RHEL 9+ only)                                 │
│                                                              │
│ Back-ends:    /etc/crypto-policies/back-ends/                │
│ Modules:      /usr/share/crypto-policies/policies/modules/   │
│                                                              │
│ After change: systemctl restart <services>                   │
│               OR reboot                                      │
│                                                              │
│ DEFAULT:      TLS 1.2+, RSA 2048+, No SHA-1                  │
│ LEGACY:       TLS 1.0+, allows weak (temporary only!)        │
│ FUTURE:       TLS 1.2+, RSA 3072+, strictest                 │
│ FIPS:         Federal compliance (requires FIPS mode)        │
└──────────────────────────────────────────────────────────────┘

⚠️ RHEL 7 doesn't have crypto-policies (manual config only)
✅ DEFAULT works for 95% of environments

🧪 Hands-On Lab

Lab 12: Crypto-Policies

Understand and configure system-wide crypto-policies

• 📁 Location: labs/en_US/12-crypto-policies/
• ⏱️ Time: 25-30 minutes
• 🎯 Level: Intermediate

Chapter Navigation

Chapter 24: Let’s Encrypt & certbot

Free Public Certificates: Let’s Encrypt provides free, automated certificates for public-facing websites. Learn how to use it on RHEL with certbot.

24.1 What is Let’s Encrypt?

Let’s Encrypt is a free, automated, and open Certificate Authority.

Key Features:

• ✅ Free certificates (no cost)
• ✅ Automated issuance (via ACME protocol)
• ✅ Auto-renewal (every 60-90 days)
• ✅ Widely trusted (in all major browsers)
• ✅ Domain validation (DV certificates)

Limitations:

• ❌ Public domains only (must be internet-accessible for validation)
• ❌ 90-day validity (short-lived, requires automation)
• ❌ Domain Validation only (no Organization or Extended Validation)
• ❌ No wildcard with HTTP-01 (requires DNS-01 challenge)

24.2 Let’s Encrypt on RHEL: Public ACME and Native Alternatives

Method 1: certbot (Traditional)

⚠️ CRITICAL: EPEL Required

certbot is NOT available in official RHEL repositories.

It requires EPEL (Extra Packages for Enterprise Linux), a community-maintained repository that is NOT officially supported by Red Hat.

For Enterprise Production Environments:

• Consider FreeIPA with certmonger (Chapter 19)
• Or commercial CA with certmonger (Chapter 22)
• Or manual certificate management

EPEL is suitable for:

• Development/testing environments
• Small deployments where EPEL risk is acceptable
• Situations where free certificates outweigh support concerns

certbot Tool:

• Full automation
• Apache/NGINX plugins
• Automatic configuration
• Renewal timers
• ⚠️ Requires EPEL

Method 2: certmonger for Internal/Private CAs

Native RHEL Solution:

• ✅ No EPEL needed
• ✅ Red Hat supported
• ✅ Best fit for FreeIPA / IdM and local CA workflows
• ⏸️ Manual web server config (no Apache/NGINX plugins)
• ❌ Not a direct replacement for certbot with public Let’s Encrypt

We’ll use certbot for public Let’s Encrypt and certmonger for native internal workflows.

24.3 certbot Installation

RHEL 7

#============================================#
# INSTALL CERTBOT ON RHEL 7 (REQUIRES EPEL!)
#============================================#

# ⚠️ WARNING: Enabling third-party repository

# Step 1: Enable EPEL
sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm -y

# Verify EPEL enabled
yum repolist | grep epel

# Step 2: Install certbot
sudo yum install certbot python2-certbot-apache python2-certbot-nginx -y

# Verify
certbot --version

RHEL 8

#============================================#
# INSTALL CERTBOT ON RHEL 8 (REQUIRES EPEL!)
#============================================#

# ⚠️ WARNING: Enabling third-party repository

# Step 1: Enable EPEL
sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm -y

# Or if you have subscription:
sudo dnf install epel-release -y

# Step 2: Install certbot
sudo dnf install certbot python3-certbot-apache python3-certbot-nginx -y

# Verify
certbot --version

RHEL 9/10

#============================================#
# INSTALL CERTBOT ON RHEL 9/10 (REQUIRES EPEL!)
#============================================#

# ⚠️ WARNING: Enabling third-party repository

# Step 1: Enable EPEL
sudo dnf install epel-release -y

# Step 2: Install certbot
sudo dnf install certbot python3-certbot-apache python3-certbot-nginx -y

# Verify
certbot --version

Remember: EPEL is community-supported. For enterprise production, consider FreeIPA + certmonger (native RHEL solution).

24.4 certbot Usage - Apache

Automatic Apache Configuration

#============================================#
# CERTBOT WITH APACHE (AUTOMATED!)
#============================================#

# Prerequisites:
# - Apache installed and running
# - Port 80 accessible from internet
# - Domain resolves to this server
# - EPEL repository enabled

# Obtain certificate and auto-configure Apache
sudo certbot --apache -d www.example.com -d example.com

# certbot will:
# 1. Generate certificate from Let's Encrypt
# 2. Automatically configure Apache SSL
# 3. Set up HTTP→HTTPS redirect
# 4. Configure auto-renewal

# Interactive prompts:
# - Email address (for renewal notices)
# - Agree to ToS
# - Redirect HTTP to HTTPS? (choose yes)

# Non-interactive (automation):
sudo certbot --apache \
  -d www.example.com \
  -d example.com \
  --non-interactive \
  --agree-tos \
  --email admin@example.com \
  --redirect

# Certificate location:
# /etc/letsencrypt/live/www.example.com/fullchain.pem
# /etc/letsencrypt/live/www.example.com/privkey.pem

24.5 certbot Usage - NGINX

Automatic NGINX Configuration

#============================================#
# CERTBOT WITH NGINX (AUTOMATED!)
#============================================#

# Prerequisites:
# - NGINX installed and running
# - Port 80 accessible
# - Domain resolves to server

# Obtain and configure
sudo certbot --nginx -d api.example.com

# Non-interactive
sudo certbot --nginx \
  -d api.example.com \
  --non-interactive \
  --agree-tos \
  --email admin@example.com \
  --redirect

# certbot updates NGINX config automatically!
# No manual SSL configuration needed

24.6 certbot Manual Mode (Standalone)

Without Web Server Plugin

#============================================#
# CERTBOT STANDALONE (NO PLUGIN)
#============================================#

# Use when:
# - Web server not Apache/NGINX
# - Want manual control over config
# - Using custom web server

# Obtain certificate only (doesn't configure server)
sudo certbot certonly --standalone \
  -d app.example.com \
  --non-interactive \
  --agree-tos \
  --email admin@example.com

# Certificate saved to:
# /etc/letsencrypt/live/app.example.com/fullchain.pem
# /etc/letsencrypt/live/app.example.com/privkey.pem

# Manually configure your service to use it
# Apache example:
# SSLCertificateFile /etc/letsencrypt/live/app.example.com/fullchain.pem
# SSLCertificateKeyFile /etc/letsencrypt/live/app.example.com/privkey.pem