[Yandex Cloud documentation](../../index.md) > [Yandex Audit Trails](../index.md) > [Tutorials](index.md) > Exporting audit logs to SIEM systems > Uploading audit logs to Splunk SIEM

# Uploading audit logs to Splunk SIEM

Create a trail to upload management event audit logs of resources in an individual folder to an Yandex Object Storage bucket with encryption enabled. Then configure continuous log delivery to SIEM Splunk.

The solution described in the tutorial follows the steps below:
1. A [trail](../concepts/trail.md) uploads logs to an Object Storage bucket.
1. The bucket is mounted as part of an intermediate VM's [filesystem](https://en.wikipedia.org/wiki/Filesystem_in_Userspace).
1. The intermediate VM runs a script that pulls logs from the bucket on a schedule and pushes them to Splunk.

To configure delivery of audit log files from a bucket to Splunk:

1. [Get your cloud ready](#before-begin).
1. [Set up your environment](#prepare-environment).
1. [Assign the required roles to your service account](#add-roles).
1. [Create a trail](#create-trail).
1. [Set up Splunk for import](#prepare-splunk).
1. [Enable egress NAT for the subnet with the intermediate VM](#enable-nat).
1. [Create an intermediate VM](#create-vm).
1. [Visualize data in Splunk](#splunk-visualization).

Some steps are completed in Terraform.

If you no longer need the resources you created, [delete them](#clear-out).

## Getting started {#before-begin}

If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../cli/quickstart.md#install).

The folder used by default is the one specified when [creating](../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id <folder_ID>` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

Sign up for Yandex Cloud and create a [billing account](../../billing/concepts/billing-account.md):
1. Navigate to the [management console](https://console.yandex.cloud) and log in to Yandex Cloud or create a new account.
1. On the **[Yandex Cloud Billing](https://center.yandex.cloud/billing/accounts)** page, make sure you have a billing account linked and it has the `ACTIVE` or `TRIAL_ACTIVE` [status](../../billing/concepts/billing-account-statuses.md). If you do not have a billing account, [create one](../../billing/quickstart/index.md) and [link](../../billing/operations/pin-cloud.md) a cloud to it.

If you have an active billing account, you can create or select a [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) for your infrastructure on the [cloud page](https://console.yandex.cloud/cloud).

[Learn more about clouds and folders here](../../resource-manager/concepts/resources-hierarchy.md).

To complete the tutorial, a Splunk instance must be available to the intermediate VM on port `8080`.

### Required paid resources {#paid-resources}

The infrastructure support cost includes:

* Fee for using VM instances (see [Compute Cloud pricing](../../compute/pricing.md)).
* Fee for storing data in a bucket (see [Object Storage pricing](../../storage/pricing.md#prices-storage)).
* Fee for operations with data (see [Object Storage pricing](../../storage/pricing.md#prices-operations)).
* Fee for using KMS keys (see [Key Management Service pricing](../../kms/pricing.md#prices)).

## Set up your environment {#prepare-environment}

### Create a new bucket to use for uploading audit logs {#create-backet}

{% list tabs group=instructions %}

- Management console {#console}

  1. In the [management console](https://console.yandex.cloud), select the folder where you want to create a [bucket](../../storage/concepts/bucket.md).
  1. Navigate to **Object Storage**.
  1. Click **Create bucket**.
  1. On the bucket creation page:
      1. Enter a name for the bucket by following the [naming conventions](../../storage/concepts/bucket.md#naming).

         By default, a bucket with a dot in the name is only available over HTTP. To provide HTTPS support for your bucket, [upload your own security certificate](../../storage/operations/hosting/certificate.md) to Object Storage.

      1. Limit the maximum bucket size, if required.

         If the value is `0`, the size is not limited. It is similar to the enabled **No limit** option.

      1. In the **Read objects**, **Read object list**, and **Read settings** fields, select `With authorization`.
      1. Select the default [storage class](../../storage/concepts/storage-class.md).
      1. Click **Create bucket** to complete the operation.

{% endlist %}

### Create an encryption key in Key Management Service {#create-key}

{% list tabs group=instructions %}

- Management console {#console}

  1. In the [management console](https://console.yandex.cloud), select the folder containing your bucket.
  1. Navigate to **Key Management Service**.
  1. Click **Create key** and set the key attributes:

     * Any name and optional description.
     * Encryption algorithm, e.g., AES-256.
     * [Rotation](../../kms/concepts/index.md#rotation) period (how often to change key versions).
     * Click **Create**.

  When creating a key, you create its first version; to open a page with its attributes, click the key in the list.

{% endlist %}

### Enable bucket encryption {#backet-encoding}

{% list tabs group=instructions %}

- Management console {#console}

  1. In the [management console](https://console.yandex.cloud), select the bucket you want to configure encryption for.
  1. In the left-hand panel, select **Security**.
  1. Open the **Encryption** tab.
  1. In the **KMS Key** field, select an existing key or create a new one:

     1. If the folder does not contain any keys yet, click **Create key**. If the folder contains keys but they are not suitable, click **Create**.
     1. Enter a name for the key.
     1. Select an encryption algorithm and a rotation period.
     1. Click **Create**.

  1. Click **Save**.

{% endlist %}

### Create a service account {#create-sa}

{% list tabs group=instructions %}

- Management console {#console}

  1. In the [management console](https://console.yandex.cloud), select the folder where you want to create a service account.
  1. Navigate to **Identity and Access Management**.
  1. Click **Create service account**.
  1. Enter a name for the service account. The naming requirements are as follows:

      * Length: between 3 and 63 characters.
      * It can only contain lowercase Latin letters, numbers, and hyphens.
      * It must start with a letter and cannot end with a hyphen.

  1. Click **Create**.

{% endlist %}

## Assign the required roles to your service account {#add-roles}

{% list tabs group=instructions %}

- CLI {#cli}

  1. Assign the [audit-trails.viewer](../security/index.md#roles) role to the folder whose resources will be polled for audit logs:
     
      ```
      yc resource-manager folder add-access-binding \
      --role audit-trails.viewer \
      --id <folder_ID> \
      --service-account-id <service_account_ID>
      ```
  
      Where:
  
      * `--role`: Role to assign.
      * `--id`: ID of the folder from which audit logs will be collected.
      * `--service-account-id`: Service account ID.
  
  1. Assign the [storage.uploader](../../storage/security/index.md#storage-uploader) role to the folder to host the trail:
     
      ```
      yc resource-manager folder add-access-binding \
      --role storage.uploader \
      --id <folder_ID> \
      --service-account-id <service_account_ID>
      ```
  
      Where:
  
      * `--role`: Role to assign.
      * `--id`: ID of the folder to host the trail.
      * `--service-account-id`: Service account ID.
  
  1. Assign the [kms.keys.encrypterDecrypter](../../kms/security/index.md#service) role to the encryption key:
  
      ```
      yc kms symmetric-key add-access-binding \
      --role kms.keys.encrypterDecrypter \
      --id <KMS_key_ID> \
      --service-account-id <service_account_ID>
      ```
  
      Where:
  
      * `--role`: Role to assign.
      * `--id`: KMS key ID.
      * `--service-account-id`: Service account ID.
  
{% endlist %}

## Create a trail {#create-trail}

To create the trail, make sure you have the following roles:

   * `iam.serviceAccounts.user` for the service account.
   * `audit-trails.editor` for the folder to host the trail.
   * `audit-trails.viewer` for the folder from which audit logs will be collected.
   * `storage.viewer` for the bucket or folder.

{% list tabs group=instructions %}

- Management console {#console}

  1. In the [management console](https://console.yandex.cloud), select the folder where you want to create the trail.
  1. Navigate to **Audit Trails**.
  1. Click **Create trail** and specify:

     * **Name**: Name of the new trail.
     * **Description**: Trail description (optional).

  1. Under **Destination**, configure the destination object:

     * **Destination**: `Object Storage`.
     * **Bucket**: Name of the [bucket](../../storage/operations/buckets/create.md) to which you want to upload audit logs.
     * **Object prefix**: Optional parameter used in the [full name](../concepts/format.md#log-file-name) of the audit log file.
  
     {% note info %}
     
     Use a [prefix](../../storage/concepts/object.md#key) to store audit logs and third-party data in the same bucket. Do not use the same prefix for logs and other bucket objects because that may cause logs and third-party objects to overwrite each other.
     
     {% endnote %}

      * **Encryption key**: Specify the encryption key the bucket is [encrypted](../../storage/concepts/encryption.md) with.
  
  1. Under **Service account**, select the service account that the trail will use to upload audit log files to the bucket.

  1. Under **Collecting management events**, configure the collection of management event audit logs:

     * **Collecting events**: Select `Enabled`.
     * **Resource**: Select `Folder`.
     * **Folder**: Automatically populated field containing the name of the current folder.

  1. Under **Collecting data events**, select `Disabled` in the **Collecting events** field.

  1. Click **Create**.

  {% note warning %}
  
  The solution will delete the logs from the bucket after they are exported to Splunk. If you need to keep the logs in the bucket, create a separate bucket and trail.
  
  {% endnote %}

{% endlist %}

## Set up Splunk for import {#prepare-splunk}

Enable `HTTPEventCollector` and follow this [guide](https://docs.splunk.com/Documentation/SplunkCloud/8.2.2105/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_Splunk_Cloud_Platform) to get the `Event Collector` token.

## Set up a NAT gateway for the subnet with the intermediate VM {#enable-nat}

{% list tabs group=instructions %}

- Management console {#console}

  1. Create a NAT gateway:
      1. In the [management console](https://console.yandex.cloud), select the folder containing the subnet for the intermediate VM.
      1. Navigate to **Virtual Private Cloud**.
      1. In the left-hand panel, select **Gateways**.
      1. Click **Create**.
      1. Enter a name for the gateway. The naming requirements are as follows:

          * Length: between 3 and 63 characters.
          * It can only contain lowercase Latin letters, numbers, and hyphens.
          * It must start with a letter and cannot end with a hyphen.
   
      1. The default gateway type is `Egress NAT`.
      1. Click **Save**.
  1. Create a route table:
      1. In the left-hand panel, select ![image](../../_assets/console-icons/route.svg) **Routing tables**.
      1. Click **Create** to [add](../../vpc/operations/static-route-create.md) a new table, or select an existing one.
      1. Click **Add**.
      1. In the window that opens, select `Gateway` in the **Next hop** field.
      1. In the **Gateway** field, select the NAT gateway you created. The destination prefix will apply automatically.
      1. Click **Add**.
      1. Click **Save**. 
  1. Link the route table to the subnet where you want to deploy the intermediate VM, to forward its traffic via the NAT gateway:
      1. In the left-hand panel, select ![image](../../_assets/console-icons/nodes-right.svg) **Subnets**.
      1. In the row with the subnet, click ![image](../../_assets/console-icons/ellipsis.svg).
      1. In the menu that opens, select **Link routing table**.
      1. In the window that opens, select your route table from the list.
      1. Click **Link**.

{% endlist %}

## Create a VM for continuous log delivery to Splunk {#create-vm}

{% list tabs group=instructions %}

- Terraform {#tf}

  1. If you do not have Terraform yet, [install it and configure the Yandex Cloud provider](../../tutorials/infrastructure-management/terraform-quickstart.md#install-terraform).
     
     
     To manage infrastructure using Terraform under a service account or user accounts (a Yandex account, a federated account, or a local user), [authenticate](../../terraform/authentication.md) using the appropriate method.
  1. Clone a [repository](https://github.com/yandex-cloud-examples/yc-export-auditlogs-to-splunk) from [Yandex Cloud Security Solution Library](https://github.com/yandex-cloud-examples/yc-security-solutions-library):
  
     ```
     git clone https://github.com/yandex-cloud-examples/yc-export-auditlogs-to-splunk.git
     ```
  
  1. Create a subfolder in `/yc-export-auditlogs-to-splunk/terraform/` and go there.
  1. Create a configuration file to invoke the `yc-splunk-trail` module:
  
     ```
     module "yc-splunk-trail" {
        source = "../modules/yc-splunk-trail/"
        folder_id = <folder_ID>
        splunk_token = <Event_Collector_token>
        splunk_server = <your_Splunk_server_address>:8088 
        bucket_name = <bucket_name>
        bucket_folder = <name_of_root_folder_in_bucket>
        sa_id = <service_account_ID>
        coi_subnet_id = <subnet_ID>
     }
     ```
     Where:
  
     * `folder_id`: Folder ID.
     * `splunk_token`: Event Collector token retrieved from Splunk.
     * `splunk_server`: Address of your Splunk server as `https://<host_name_or_address>`.
     * `bucket_name`: Bucket name.
     * `bucket_folder`: Name of the root folder in the bucket.
     * `sa_id`: Service account ID.
     * `coi_subnet_id`: ID of the subnet where you set up the NAT gateway.
  
  1. Make sure the configuration files are correct:
  
     ```
     terraform plan
     ```
  
     If the configuration is correct, the terminal will display a list of the resources and their settings. Otherwise, Terraform will show any detected errors.
  
  1. Deploy the cloud resources.
     1. If the configuration is correct, run this command:
  
        ```
        terraform apply
        ```
  
     1. Confirm resource creation: enter `yes` in the terminal and press **Enter**.

{% endlist %}

## Visualize data in Splunk {#splunk-visualization}

1. Go to Splunk and search for the events created by the service account:

   ```
   index="main" authentication.subject_type="SERVICE_ACCOUNT" | stats count by event_type
   ```

1. To visualize the results, go to the **Visualization** tab and select a suitable format (`LineChart`, `PieChart`, etc.):

   Data visualization example:

   ![image](../../_assets/audit-trails/tutorials/splunk-dashboard.png)

## How to delete the resources you created {#clear-out}

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

1. To delete the resources created with Terraform:

   1. Run this command:
      ```
      terraform destroy
      ```

      {% note warning %}

      Terraform will delete all the resources that you created in the current configuration, such as networks, subnets, virtual machines, and so on.

      {% endnote %}


      After the command is executed, the terminal will display a list of resources to be deleted.

   1. To confirm deletion, type `yes` and press **Enter**.

1. [Delete the Object Storage bucket](../../storage/operations/buckets/delete.md).

1. [Delete the Key Management Service key](../../kms/operations/key.md#delete).

1. [Delete the route table](../../vpc/operations/delete-route-table.md).

1. [Delete the NAT gateway](../../vpc/operations/delete-nat-gateway.md).