[Yandex Cloud documentation](../../../index.md) > [Yandex MetaData Hub](../../index.md) > Apache Hive™ Metastore > Step-by-step guides > Transferring logs to Cloud Logging

# Transferring a Apache Hive™ Metastore cluster's logs to Yandex Cloud Logging

You can set up regular collection of Apache Hive™ Metastore cluster performance logs. Logs will be delivered to a [log group](../../../logging/concepts/log-group.md) in Yandex Cloud Logging. You can choose between these two types of log groups:

* Default log group of the cluster folder.
* Custom log group.

## Getting started {#before-you-begin}

1. [Create a service account](../../../iam/operations/sa/create.md) named `metastore-logging-sa`.
1. [Assign the `managed-metastore.integrationProvider` role](../../../iam/operations/sa/assign-role-for-sa.md) to the service account.
1. [Set up a NAT gateway](../../../vpc/operations/create-nat-gateway.md) in the subnet the cluster will connect to. This will enable the cluster to work with Yandex Cloud services.
1. [Configure the security group](configure-security-group.md).

## Transferring data to the default log group {#default}

1. Create a Apache Hive™ Metastore cluster:

   {% list tabs group=instructions %}

   - Management console {#console}

      1. In the [management console](https://console.yandex.cloud), select the relevant folder.
      1. [Navigate](../../../console/operations/select-service.md#select-service) to **Yandex MetaData Hub**.
      1. In the left-hand panel, select ![image](../../../_assets/console-icons/database.svg) **Metastore**.
      1. Click **Create cluster**.
      1. Specify `metastore-cluster` as the cluster name.
      1. In the **Service account** field, select `metastore-logging-sa`.
      1. Under **Network settings**, select the network and subnet with a NAT gateway configured. Specify the security group you configured.
      1. Under **Logging**:

         1. Enable **Write logs**.
         1. To write logs to the default log group, select **Folder** in the **Destination** field.
         1. Specify the folder whose log group you want to use.
         1. Select the minimum logging level.

            The execution log will contain logs of this level or higher. The available levels are `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, and `FATAL`. The default is `INFO`.

      1. Click **Create**.

   - CLI {#cli}

      If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../../cli/quickstart.md#install).

      The folder used by default is the one specified when [creating](../../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id <folder_ID>` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

      Run this command:

      ```bash
      yc managed-metastore cluster create \
         --name metastore-cluster \
         --service-account-id <service_account_ID> \
         --version <Apache Hive™ Metastore_version> \
         --subnet-ids <subnet_IDs> \
         --security-group-ids <security_group_IDs> \
         --resource-preset-id <ID_of_computing_resources> \
         --log-enabled \
         --log-folder-id <folder_ID> \
         --log-min-level <logging_level>
      ```

      Where:

      * `--service-account-id`: ID of the `metastore-logging-sa` service account you created [earlier](#before-you-begin).
      * `--subnet-ids`: ID of the subnet where you set up the NAT gateway.
      * `--security-group-ids`: ID of the security group you configured [earlier](#before-you-begin).
      * `--log-folder-id`: Specify the ID of the folder whose log group you want to use.
      * `--log-min-level`: Logging level. The execution log will contain logs of this level or higher. The available levels are `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, and `FATAL`. The default is `INFO`.

      [Learn more about creating a cluster](cluster-create.md).

   - REST API {#api}

        1. [Get an IAM token for API authentication](../../api-ref/authentication.md) and put it into an environment variable:

            ```bash
            export IAM_TOKEN="<IAM_token>"
            ```

        1. Create a file named `body.json` and paste the following code into it:

            ```json
            {
              "folderId": "<folder_ID>",
              "name": "metastore-cluster",
              "version": "<Apache Hive™ Metastore_version>",
              "configSpec": {
                "resources": {
                "resourcePresetId": "<resource_configuration_ID>"
                }
              },
              "serviceAccountId": "<service_account_ID>",
              "logging": {
                "enabled": true,
                "folderId": "<folder_ID>",
                "minLevel": "<logging_level>"
              },
              "network": {
                "subnetIds": [ "<list_of_subnet_IDs>" ],
                "securityGroupIds": [ "<list_of_security_group_IDs>" ]
              },
            }
            ```

            Where:

            * `serviceAccountId`: ID of the `metastore-logging-sa` service account you created [earlier](#before-you-begin).
            * `logging.folderId`: Specify the ID of the folder whose log group you want to use.
            * `logging.minLevel`: Logging level. The execution log will contain logs of this level or higher. The available levels are `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, and `FATAL`. The default is `INFO`.
            * `network.subnetIds`: ID of the subnet where you set up the NAT gateway.
            * `network.securityGroupIds`: ID of the security group you configured [earlier](#before-you-begin).

            [Learn more about creating a cluster](cluster-create.md).

        1. Call the [Cluster.Create](../../api-ref/Cluster/create.md) method, e.g., via the following [cURL](https://curl.se/) request:

            ```bash
            curl \
                --request POST \
                --header "Authorization: Bearer $IAM_TOKEN" \
                --url 'https://metastore.api.cloud.yandex.net/managed-metastore/v1/clusters' \
                --data '@body.json'
            ```

        1. View the [server response](../../api-ref/Cluster/create.md#yandex.cloud.operation.Operation) to make sure your request was successful.

   - gRPC API {#grpc-api}

        1. [Get an IAM token for API authentication](../../api-ref/authentication.md) and place it in an environment variable:

            ```bash
            export IAM_TOKEN="<IAM_token>"
            ```

        1. Clone the [cloudapi](https://github.com/yandex-cloud/cloudapi) repository:
           
           ```bash
           cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
           ```
           
           Below, we assume that the repository contents reside in the `~/cloudapi/` directory.

        1. Create a file named `body.json` and paste the following code into it:

            ```json
            {
              "folder_id": "<folder_ID>",
              "name": "metastore-cluster",
              "version": "<Apache Hive™ Metastore_version>",
              "config_spec": {
                "resources": {
                  "resource_preset_id": "<resource_configuration_ID>"
                }
              },
              "service_account_id": "<service_account_ID>",
              "logging": {
                "enabled": true,
                "folder_id": "<folder_ID>",
                "min_level": "<logging_level>"
              },
              "network": {
                "subnet_ids": [ "<list_of_subnet_IDs>" ],
                "security_group_ids": [ "<list_of_security_group_IDs>" ]
              },
            }
            ```

            Where:

            * `service_account_id`: ID of the `metastore-logging-sa` service account you created [earlier](#before-you-begin).
            * `logging.folder_id`: Specify the ID of the folder whose log group you want to use.
            * `logging.min_level`: Logging level. The execution log will contain logs of this level or higher. The available levels are `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, and `FATAL`. The default is `INFO`.
            * `network.subnet_ids`: ID of the subnet where you set up the NAT gateway.
            * `network.security_group_ids`: ID of the security group you configured [earlier](#before-you-begin).

            [Learn more about creating a cluster](cluster-create.md).

        1. Call the [ClusterService.Create](../../api-ref/grpc/Cluster/create.md) method, e.g., via the following [gRPCurl](https://github.com/fullstorydev/grpcurl) request:

            ```bash
            grpcurl \
                -format json \
                -import-path ~/cloudapi/ \
                -import-path ~/cloudapi/third_party/googleapis/ \
                -proto ~/cloudapi/yandex/cloud/metastore/v1/cluster_service.proto \
                -rpc-header "Authorization: Bearer $IAM_TOKEN" \
                -d @ \
                metastore.api.cloud.yandex.net:443 \
                yandex.cloud.metastore.v1.ClusterService.Create \
                < body.json
            ```

        1. Check the [server response](../../api-ref/grpc/Cluster/create.md#yandex.cloud.operation.Operation) to make sure your request was successful.

   {% endlist %}

1. Test the transfer of cluster logs to the log group:

   {% list tabs group=instructions %}

   - Management console {#console}

      1. In the management console, navigate to the relevant folder.
      1. [Navigate](../../../console/operations/select-service.md#select-service) to **Cloud Logging**.
      1. Click the row with the `default` log group.

      The page that opens will show the log group entries.

   - CLI {#cli}

      If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../../cli/quickstart.md#install).

      The folder used by default is the one specified when [creating](../../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id <folder_ID>` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

      To view the entries in JSON format, run this command:

      ```bash
      yc logging read --group-name=default --format=json
      ```

      Result:

      ```text
      [
        {
          "uid": "23:6d******:3",
          "resource": {
            "type": "managed-metastore.cluster",
            "id": "c9qcv46p3g3a********"
          },
          "timestamp": "2024-11-18T09:45:23.092484Z",
          "ingested_at": "2024-11-18T09:45:24Z",
          "saved_at": "2024-11-18T09:45:25.559195Z",
          "level": "INFO",
          "message": "ugi=metastore\tip=10.253.89.135\tcmd=Done cleaning up thread local RawStore\t",
          "json_payload": {
            "instance": "metastore-server-5f9c7ff***-*****",
            "logger": "org.apache.hadoop.hive.metastore.HiveMetaStore.audit",
            "resource_id": "c9qcv46p3g3a********",
            "stream_name": "metastore-server",
            "thread": "pool-6-thread-79"
          },
          "stream_name": "metastore-server"
        }
      ]
      ```

   - API {#api}

      To view log group entries, use the [LogReadingService.Read](../../../logging/api-ref/grpc/LogReading/read.md) gRPC API call.

   {% endlist %}

   For more information, see [Reading records](../../../logging/operations/read-logs.md).

## Sending data to a custom log group {#custom}

1. [Create a log group](../../../logging/operations/create-group.md) named `metastore-log-group`.
1. Create a Apache Hive™ Metastore cluster:

   {% list tabs group=instructions %}

   - Management console {#console}

      1. In the [management console](https://console.yandex.cloud), select the relevant folder.
      1. [Navigate](../../../console/operations/select-service.md#select-service) to **Yandex MetaData Hub**.
      1. In the left-hand panel, select ![image](../../../_assets/console-icons/database.svg) **Metastore**.
      1. Click **Create cluster**.
      1. Specify `metastore-cluster` as the cluster name.
      1. In the **Service account** field, select `metastore-logging-sa`.
      1. Under **Network settings**, select the network and subnet with a NAT gateway configured. Specify the security group you configured.
      1. Under **Logging**:

         1. Enable **Write logs**.
         1. To write logs to a custom log group, select **Log group** in the **Destination** field.
         1. Specify the `metastore-log-group` log group.
         1. Select the minimum logging level.

            The execution log will contain logs of this level or higher. The available levels are `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, and `FATAL`. The default is `INFO`.

      1. Click **Create**.

   - CLI {#cli}

      If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../../cli/quickstart.md#install).

      The folder used by default is the one specified when [creating](../../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id <folder_ID>` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

      Run this command:

      ```bash
      yc managed-metastore cluster create \
         --name metastore-cluster \
         --service-account-id <service_account_ID> \
         --version <Apache Hive™ Metastore_version> \
         --subnet-ids <subnet_IDs> \
         --security-group-ids <security_group_IDs> \
         --resource-preset-id <ID_of_computing_resources> \
         --log-enabled \
         --log-group-id <log_group_ID> \
         --log-min-level <logging_level>
      ```

      Where:

      * `--service-account-id`: ID of the `metastore-logging-sa` service account you created [earlier](#before-you-begin).
      * `--subnet-ids`: ID of the subnet where you set up the NAT gateway.
      * `--security-group-ids`: ID of the security group you configured [earlier](#before-you-begin).
      * `--log-group-id`: `metastore-log-group` log group ID.
      * `--log-min-level`: Logging level. The execution log will contain logs of this level or higher. The available levels are `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, and `FATAL`. The default is `INFO`.

      [Learn more about creating a cluster](cluster-create.md).

   - REST API {#api}

        1. [Get an IAM token for API authentication](../../api-ref/authentication.md) and put it into an environment variable:

            ```bash
            export IAM_TOKEN="<IAM_token>"
            ```

        1. Create a file named `body.json` and paste the following code into it:

            ```json
            {
              "folderId": "<folder_ID>",
              "name": "metastore-cluster",
              "version": "<Apache Hive™ Metastore_version>",
              "configSpec": {
                "resources": {
                "resourcePresetId": "<resource_configuration_ID>"
                }
              },
              "serviceAccountId": "<service_account_ID>",
              "logging": {
                "enabled": true,
                "logGroupId": "<log_group_ID>",
                "minLevel": "<logging_level>"
              },
              "network": {
                "subnetIds": [ "<list_of_subnet_IDs>" ],
                "securityGroupIds": [ "<list_of_security_group_IDs>" ]
              },
            }
            ```

            Where:

            * `serviceAccountId`: ID of the `metastore-logging-sa` service account you created [earlier](#before-you-begin).
            * `logging.logGroupId`: `metastore-log-group` log group ID.
            * `logging.minLevel`: Logging level. The execution log will contain logs of this level or higher. The available levels are `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, and `FATAL`. The default is `INFO`.
            * `network.subnetIds`: ID of the subnet where you set up the NAT gateway.
            * `network.securityGroupIds`: ID of the security group you configured [earlier](#before-you-begin).

            [Learn more about creating a cluster](cluster-create.md).

        1. Call the [Cluster.Create](../../api-ref/Cluster/create.md) method, e.g., via the following [cURL](https://curl.se/) request:

            ```bash
            curl \
                --request POST \
                --header "Authorization: Bearer $IAM_TOKEN" \
                --url 'https://metastore.api.cloud.yandex.net/managed-metastore/v1/clusters' \
                --data '@body.json'
            ```

        1. View the [server response](../../api-ref/Cluster/create.md#yandex.cloud.operation.Operation) to make sure your request was successful.

   - gRPC API {#grpc-api}

        1. [Get an IAM token for API authentication](../../api-ref/authentication.md) and place it in an environment variable:

            ```bash
            export IAM_TOKEN="<IAM_token>"
            ```

        1. Clone the [cloudapi](https://github.com/yandex-cloud/cloudapi) repository:
           
           ```bash
           cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
           ```
           
           Below, we assume that the repository contents reside in the `~/cloudapi/` directory.

        1. Create a file named `body.json` and paste the following code into it:

            ```json
            {
              "folder_id": "<folder_ID>",
              "name": "metastore-cluster",
              "version": "<Apache Hive™ Metastore_version>",
              "config_spec": {
                "resources": {
                  "resource_preset_id": "<resource_configuration_ID>"
                }
              },
              "service_account_id": "<service_account_ID>",
              "logging": {
                "enabled": true,
                "log_group_id": "<log_group_ID>",
                "min_level": "<logging_level>"
              },
              "network": {
                "subnet_ids": [ "<list_of_subnet_IDs>" ],
                "security_group_ids": [ "<list_of_security_group_IDs>" ]
              },
            }
            ```

            Where:

            * `service_account_id`: ID of the `metastore-logging-sa` service account you created [earlier](#before-you-begin).
            * `logging.log_group_id`: `metastore-log-group` log group ID.
            * `logging.min_level`: Logging level. The execution log will contain logs of this level or higher. The available levels are `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, and `FATAL`. The default is `INFO`.
            * `network.subnet_ids`: ID of the subnet where you set up the NAT gateway.
            * `network.security_group_ids`: ID of the security group you configured [earlier](#before-you-begin).

            [Learn more about creating a cluster](cluster-create.md).

        1. Call the [ClusterService.Create](../../api-ref/grpc/Cluster/create.md) method, e.g., via the following [gRPCurl](https://github.com/fullstorydev/grpcurl) request:

            ```bash
            grpcurl \
                -format json \
                -import-path ~/cloudapi/ \
                -import-path ~/cloudapi/third_party/googleapis/ \
                -proto ~/cloudapi/yandex/cloud/metastore/v1/cluster_service.proto \
                -rpc-header "Authorization: Bearer $IAM_TOKEN" \
                -d @ \
                metastore.api.cloud.yandex.net:443 \
                yandex.cloud.metastore.v1.ClusterService.Create \
                < body.json
            ```

        1. Check the [server response](../../api-ref/grpc/Cluster/create.md#yandex.cloud.operation.Operation) to make sure your request was successful.

   {% endlist %}

1. Test the transfer of cluster logs to the log group:

   {% list tabs group=instructions %}

   - Management console {#console}

      1. In the management console, navigate to the relevant folder.
      1. [Navigate](../../../console/operations/select-service.md#select-service) to **Cloud Logging**.
      1. Click the row with the `metastore-log-group` log group.

      The page that opens will show the records.

   - CLI {#cli}

      If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../../cli/quickstart.md#install).

      The folder used by default is the one specified when [creating](../../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id <folder_ID>` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

      To view the entries in JSON format, run this command:

      ```bash
      yc logging read --group-name=metastore-log-group --format=json
      ```

      Result:

      ```text
      [
        {
          "uid": "23:6d******:3",
          "resource": {
            "type": "managed-metastore.cluster",
            "id": "c9qcv46p3g3a********"
          },
          "timestamp": "2024-11-18T09:45:23.092484Z",
          "ingested_at": "2024-11-18T09:45:24Z",
          "saved_at": "2024-11-18T09:45:25.559195Z",
          "level": "INFO",
          "message": "ugi=metastore\tip=10.253.89.135\tcmd=Done cleaning up thread local RawStore\t",
          "json_payload": {
            "instance": "metastore-server-5f9c7ff***-*****",
            "logger": "org.apache.hadoop.hive.metastore.HiveMetaStore.audit",
            "resource_id": "c9qcv46p3g3a********",
            "stream_name": "metastore-server",
            "thread": "pool-6-thread-79"
          },
          "stream_name": "metastore-server"
        }
      ]
      ```

   - API {#api}

      To view log group entries, use the [LogReadingService.Read](../../../logging/api-ref/grpc/LogReading/read.md) gRPC API call.

   {% endlist %}

   For more information, see [Reading records](../../../logging/operations/read-logs.md).

_Apache® and [Apache Hive™](https://hive.apache.org/) are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries._