# Using the yandex-lemmer plugin in Managed Service for OpenSearch

# Using the yandex-lemmer plugin in Yandex Managed Service for OpenSearch


With `yandex-lemmer`, you can add a filter to [improve text search in Russian](../concepts/plugins.md#yandex-lemmer) across OpenSearch documents.

To test the plugin, compare [standard search results](#search-without-yandex-lemmer) to [`yandex-lemmer` search results](#search-with-yandex-lemmer) and then [refine your filter](#improve-search).

After testing, [delete](#clear-out) the created resources if you no longer need them.


## Required paid resources {#paid-resources}

* Managed Service for OpenSearch cluster: use of computing resources, storage and backup size (see [Managed Service for OpenSearch pricing](../pricing.md)).
* Public IP addresses if public access is enabled for cluster hosts (see [Yandex Virtual Private Cloud pricing](../../vpc/pricing.md)).


## Getting started {#before-you-begin}

1. Set up your infrastructure:

    {% list tabs group=instructions %}

    * Manually {#manual}

        1. [Create a Managed Service for OpenSearch cluster](../operations/cluster-create.md#create-cluster) of your preferred configuration with the following settings:

            * `yandex-lemmer` and `analysis-icu` plugins enabled.
            * Public access to a group of hosts with the `DATA` role enabled.

            {% note info %}
            
            Public access to cluster hosts is required if you plan to connect to the cluster via the internet. This connection option is simpler and is recommended for the purposes of this guide. You can connect to non-public hosts as well but only from Yandex Cloud virtual machines located in the same cloud network as the cluster.
            
            {% endnote %}

        1. If there are security groups in your cluster, make sure they are [configured correctly](../operations/connect/index.md#configuring-security-groups) and allow connections to the cluster.

    * Using Terraform {#tf}

        1. If you do not have Terraform yet, [install it](../../tutorials/infrastructure-management/terraform-quickstart.md#install-terraform).
        1. [Get the authentication credentials](../../tutorials/infrastructure-management/terraform-quickstart.md#get-credentials). You can add them to environment variables or specify them later in the provider configuration file.
        1. [Configure and initialize a provider](../../tutorials/infrastructure-management/terraform-quickstart.md#configure-provider). There is no need to create a provider configuration file manually, you can [download it](https://github.com/yandex-cloud-examples/yc-terraform-provider-settings/blob/main/provider.tf).
        1. Place the configuration file in a separate working directory and [specify the parameter values](../../tutorials/infrastructure-management/terraform-quickstart.md#configure-provider). If you did not add the authentication credentials to environment variables, specify them in the configuration file.

        1. Download the [opensearch-yandex-lemmer.tf](https://github.com/yandex-cloud-examples/yc-opensearch-yandex-lemmer/blob/main/opensearch-yandex-lemmer.tf) configuration file to the same working directory. This file describes:

            * [Network](../../vpc/concepts/network.md#network).
            * [Subnet](../../vpc/concepts/network.md#subnet).
            * [Security group](../../vpc/concepts/security-groups.md) and rules for connection to a Managed Service for OpenSearch cluster.
            * Managed Service for OpenSearch cluster.

        1. In the `opensearch-yandex-lemmer.tf` file, specify the following variables:

            * `version`: OpenSearch version.
            * `admin_password`: OpenSearch admin password.

        1. Validate your Terraform configuration files using this command:

            ```bash
            terraform validate
            ```

            Terraform will display any configuration errors detected in your files.

        1. Create the required infrastructure:

            1. Run this command to view the planned changes:
            
               ```bash
               terraform plan
               ```
            
               If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
            
            1. If everything looks correct, apply the changes:
               1. Run this command:
            
                  ```bash
                  terraform apply
                  ```
            
               1. Confirm updating the resources.
               1. Wait for the operation to complete.

            All the required resources will be created in the specified folder. You can check resource availability and their settings in the [management console](https://console.yandex.cloud).

    {% endlist %}

1. [Install an SSL certificate](../operations/connect/index.md#ssl-certificate).

1. Check the connection to the cluster using [cURL](https://curl.haxx.se/):

    ```bash
    curl \
        --user admin:<password> \
        --cacert ~/.opensearch/root.crt \
        --request GET 'https://<FQDN_of_OpenSearch_host_with_public_access>:9200/'
    ```
    
    You can get the host FQDN with the [list of hosts in the cluster](../operations/host-groups.md#list-hosts).
    
    If the connection is successful, you will see a message like this:
    
    ```bash
    {
      "name" : "....mdb.yandexcloud.net",
      "cluster_name" : "...",
      "cluster_uuid" : "...",
      "version" : {
      "distribution" : "opensearch",
      ...
      },
      "tagline" : "The OpenSearch Project: https://opensearch.org/"
    }
    ```

## Run a search without yandex-lemmer {#search-without-yandex-lemmer}

1. Add a document to the index named `simple-index`:

    ```bash
    curl \
        --user admin:<password> \
        --cacert ~/.opensearch/root.crt \
        --header 'Content-Type: application/json' \
        --request POST 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/simple-index/_doc?pretty' \
        --data '{
                  "book": "The Night It Rained",
                  "author": "Eugenia Riley"
                }'
    ```

    `simple-index` will be created automatically when you create the document. By default, indexes use the `Standard` [built-in text analyzer](https://opensearch.org/docs/latest/analyzers/supported-analyzers/index/#built-in-analyzers).

1. Run this search:

    ```bash
    curl \
        --user admin:<password> \
        --cacert ~/.opensearch/root.crt \
        --header 'Content-Type: application/json' \
        --request GET 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/simple-index/_search?pretty' \
        --data '{
                  "query": {
                    "query_string": {
                      "query": "book: when it rains"
                    }
                  }
                }'
    ```

    The result contains the following lines:

    ```bash
    ...
    "hits" : {
      "total" : {
        "value" : 0,
        "relation" : "eq"
      },
      "max_score" : null,
      "hits" : [ ]
    }
    ...
    ```

    The book was not found because of mismatch between the source document and the search query: `it rained` vs. `it rains`.

## Run a search with yandex-lemmer {#search-with-yandex-lemmer}

1. Create an index named `lemmer-index` with the `lemmer` analyzer parameters and connect it for the `book` field:

    ```bash
    curl \
        --user admin:<password> \
        --cacert ~/.opensearch/root.crt \
        --header 'Content-Type: application/json' \
        --request PUT "https://<address_of_OpenSearch_host_with_DATA_role>:9200/lemmer-index?pretty" \
        --data '{
                  "mappings": {
                    "properties": {
                      "book": {
                        "type": "text",
                        "analyzer": "lemmer"
                      }
                    }
                  },
                  "settings": {
                    "analysis": {
                      "analyzer": {
                        "lemmer": {
                          "type": "custom",
                          "tokenizer": "standard",
                          "filter": [
                            "yandex_lemmer"
                          ]
                        }
                      }
                    }
                  }
                }'
    ```

1. Add a document to `lemmer-index`:

    ```bash
    curl \
        --user admin:<password> \
        --cacert ~/.opensearch/root.crt \
        --header 'Content-Type: application/json' \
        --request POST 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/lemmer-index/_doc?pretty' \
        --data '{
                  "book": "The Night It Rained",
                  "author": "Eugenia Riley"
                }'
    ```

1. Run this search:

    ```bash
    curl \
        --user admin:<password> \
        --cacert ~/.opensearch/root.crt \
        --header 'Content-Type: application/json' \
        --request GET 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/lemmer-index/_search?pretty' \
        --data '{
                  "query": {
                    "query_string": {
                      "query": "book: when it rains"
                    }
                  }
                }'
    ```

    The result contains the following lines:

    ```bash
    ...
    "hits" : {
      "total" : {
        "value" : 1,
        "relation" : "eq"
      },
      "max_score" : 0.9993168,
      "hits" : [
        {
          "_index" : "lemmer-index",
          "_id" : "zoGb_ZIB-3clfLPNjPbo",
          "_score" : 0.9993168,
          "_source" : {
            "book" : "The Night It Rained",
            "author" : "Eugenia Riley"
          }
        }
      ]
    }
    ...
    ```

    The book was found.

1. To see how the `lemmer` analyzer works, run this command:

    ```bash
    curl \
        --user admin:<password> \
        --cacert ~/.opensearch/root.crt \
        --header 'Content-Type: application/json' \
        --request GET 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/lemmer-index/_analyze?pretty' \
        --data '{"field":"book","text":"it rained"}'
    ```

    Result:

    ```bash
    {
      "tokens" : [
        {
          "token" : "rained",
          "start_offset" : 0,
          "end_offset" : 3,
          "type" : "<ALPHANUM>",
          "position" : 0
        },
        {
          "token" : "rain",
          "start_offset" : 0,
          "end_offset" : 3,
          "type" : "<ALPHANUM>",
          "position" : 0
        },
        {
          "token" : "it",
          "start_offset" : 4,
          "end_offset" : 9,
          "type" : "<ALPHANUM>",
          "position" : 1
        }
      ]
    }
    ```

    After being filtered through `yandex_lemmer`, the `rained` token transforms into two tokens:

    * `rained`: Source token.
    * `rain`: Root form of `rained`.

    After being filtered through `yandex_lemmer`, the `it` token remains unchanged because the source token matches the root form of the word.

## Refine your search {#improve-search}

To refine a text search, add more filters to `yandex-lemmer`:

* `nfc`: Performs [NFC normalization](https://www.unicode.org/faq/normalization.html) (text mapping), where the same letter is represented by different characters. This filter is available if the `analysis-icu` plugin is enabled.
* `lowercase`: Converts text to lower case.
* `yo_ye`: Replaces the letter `ё` with `е`. `yandex-lemmer` does not distinguish between these letters (it will find both `ёлка` and `елка`), but such a replacement allows you to improve the percentage of analyzer cache hits and save some disk space.

To use additional filters:

1. Create an index named `index-with-filters` and add to it the `yandex-lemmer` filter and additional filters for the `lemmer-improved` analyzer:

    ```bash
    curl \
        --user admin:<password> \
        --cacert ~/.opensearch/root.crt \
        --header 'Content-Type: application/json' \
        --request PUT "https://<address_of_OpenSearch_host_with_DATA_role>:9200/index-with-filters?pretty" \
        --data '{
                  "mappings": {
                    "properties": {
                      "book": {
                        "type": "text",
                        "analyzer": "lemmer-improved"
                      }
                    }
                  },
                  "settings": {
                    "analysis": {
                      "filter": {
                        "nfc": {
                          "type": "icu_transform",
                          "id": "NFC"
                        },
                        "yo_ye": {
                          "type": "pattern_replace",
                          "pattern": "ё",
                          "replacement": "е"
                        }
                      },
                      "analyzer": {
                        "lemmer-improved": {
                          "type": "custom",
                          "tokenizer": "standard",
                          "filter": [
                            "nfc",
                            "lowercase",
                            "yo_ye",
                            "yandex_lemmer"
                          ]
                        }
                      }
                    }
                  }
                }'
    ```

1. Add a document to the index:

    ```bash
    curl \
        --user admin:<password> \
        --cacert ~/.opensearch/root.crt \
        --header 'Content-Type: application/json' \
        --request POST "https://<address_of_OpenSearch_host_with_DATA_role>.mdb.yandexcloud.net:9200/index-with-filters/_doc?pretty" \
        --data '{
                  "book": "Чёрный тюльпан",
                  "author": "Александр Дюма"
                }'
    ```

    Here, the letter `и` (code 1048) and the `˘` [breve](https://en.wikipedia.org/wiki/Breve) (code 774) are used instead of `й`.

1. Run this search:

    ```bash
    curl \
        --user admin:<password> \
        --cacert ~/.opensearch/root.crt \
        --header 'Content-Type: application/json' \
        --request GET 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/index-with-filters2/_search?pretty' \
        --data '{
                  "query": {
                    "query_string": {
                      "query": "book: черный тюльпан"
                    }
                  }
                }'
    ```

    Here, the book name includes the letter `й` (code 1049).

    The result contains the following lines:

    ```bash
    ...
    "hits" : {
      "total" : {
        "value" : 1,
        "relation" : "eq"
      },
      "max_score" : 0.2876821,
      "hits" : [
        {
          "_index" : "index-with-filters2",
          "_id" : "34EO_pIB-3clfLPNvvbY",
          "_score" : 0.2876821,
          "_source" : {
            "book" : "Чёрный тюльпан",
            "author" : "Александр Дюма"
          }
        }
      ]
    }
    ...
    ```

    The book was found.

## Delete the resources you created {#clear-out}

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

{% list tabs group=instructions %}

* Manually {#manual}

    [Delete the Managed Service for OpenSearch cluster](../operations/cluster-delete.md).

* Using Terraform {#tf}

    1. In the terminal window, go to the directory containing the infrastructure plan.
    
        {% note warning %}
    
        Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.
    
        {% endnote %}
    
    1. Delete resources:
    
        1. Run this command:
    
            ```bash
            terraform destroy
            ```
    
        1. Confirm deleting the resources and wait for the operation to complete.
    
        All the resources described in the Terraform manifests will be deleted.

{% endlist %}