Using Datadog CoTerm

View recorded terminal sessions

At the beginning and end of every recorded terminal session, CoTerm displays a link to view the session in Datadog. You can also view all recorded terminal sessions.

CoTerm CLI command structure

ddcoterm [OPTIONS] [-- <COMMAND>...] [COMMAND]

Run ddcoterm --help for all options and commands.

Record a terminal session

CoTerm records terminal sessions that you can play back and review in Datadog. For your security, sensitive data (such as passwords and API keys) are automatically redacted. Any processes launched in the terminal session are recorded as events.

Launch and record an interactive terminal session

To manually launch Datadog CoTerm and record the entirety of your terminal session:

ddcoterm

When you end the session, CoTerm stops recording and sends the captured process data to Datadog.

Record the output of a command

To run an individual command and record its output:

ddcoterm -- datadog-agent status

This launches CoTerm and runs datadog-agent status. When the process completes, CoTerm stops recording and sends the captured process data to Datadog.

Automatically record a command

To configure CoTerm to automatically record all future invocations of a particular command, create a shim:

ddcoterm shim create datadog-agent

After you create a shim, restart your terminal or source your profile. (For example, run source ~/.bashrc.) If you are using a shell other than Bash or Zsh, add path/to/.ddcoterm/overrides to your PATH manually.

Protect against dangerous terminal commands

To prevent the accidental execution of designated terminal commands, you can configure CoTerm to act as a linter. For more control, you can use CoTerm with Datadog Case Management to require approval for designated commands.

Lint a command

When you try to execute a designated command (for example, kubectl scale), CoTerm can display warnings and prompt you for confirmation.

  1. Create a shim for your command: ddcoterm shim create kubectl

  2. Configure a linting rule in your .ddcoterm/config.yaml file. For details on how to configure linting in CoTerm, see CoTerm Configuration Rules.

    .ddcoterm/config.yaml

    process_config:
      commands:
        - command: "kubectl"
          lints:
            - |
              if has_arg("scale") and flags.context == nil then
                return string.format("No kubectl context specified (effective context: '%s'). It is recommended to always explicitly specify the context when running `kubectl scale`.", k8s_context)
              end          
       

With this configuration, CoTerm intercepts any kubectl scale command without a --context flag.

Command line interface. The user has run 'kubectl scale foo'. The output says 'Warning from CoTerm: No kubectl context specified (effective context: 'minikube'). It is recommended to always explicitly specify the context when running kubectl scale. Do you want to continue? (y/n)'

Require approval for commands

For even more dangerous commands, CoTerm can require explicit approval by another team member (through Case Management) before running the command.

  1. Create a shim for your command: ddcoterm shim create kubectl

  2. Configure requiring approval in your .ddcoterm/config.yaml file. For details, see CoTerm Configuration Rules.

    .ddcoterm/config.yaml

    process_config:
      commands:
        - command: "kubectl"
          rules:
            # Record and require approval for all executions of `kubectl scale` in a production context
            - rule: |
                local applicable = has_arg("scale") and k8s_context:match("prod")
                local user_message = "Proceed with caution. This command may disrupt your Kubernetes cluster setup."
                local approver_message = "Ensure that the user has documented a rollback plan before approving."
                return applicable, user_message, approver_message            
              actions: ["record", "logs", "process_info", "approval"]
       

With this configuration, when you run a kubectl scale --context prod command, CoTerm creates an approval request in Case Management. If you opt to associate the approval request with an active incident, other incident responders are automatically added as approvers. After this request is approved, your command executes.

Manually require approval

To create an approval request manually, run:

ddcoterm approve

Bypass approval

To bypass approval and run your command, set the COTERM_BREAK_GLASS environment variable.

For example:

COTERM_BREAK_GLASS=true kubectl delete foo

Further reading

PREVIEWING: gorkavicente/appsec-serverless-library-compatibility