Deleting logs from Elastic cluster

Sometimes you might need to delete data send to Elastic. For example, someone posts personal data into application logs on accident.

It's possible to delete documents, based on Kibana filter using Delete by query API. It uses Elasticsearch query and not KQL, but in Kibana we can transform KQL to Elasticsearch quite easily.

1. Filter messages in Kibana Discover - build your filter using filter fields, not KQL syntax. See the picture - it filters logs from cert-manager kubernetes namespace
2. Click on Inspect -> Request. Scroll down until you find JSON object with starting with query. Copy this part of request.
3. Verify it returns expected documents by using Search API. Go to Dev Tools from panel and search for query using Search API. For example:

   GET /cluster-fra-dev/_search
   {
       "query": {
       "bool": {
         "must": [],
         "filter": [
           {
             "range": {
               "@timestamp": {
                 "format": "strict_date_optional_time",
                 "gte": "2023-10-26T08:45:03.732Z",
                 "lte": "2023-10-26T09:15:03.732Z"
               }
             }
           },
           {
             "match_phrase": {
               "kubernetes.namespace": "cert-manager"
             }
           }
         ],
         "should": [],
         "must_not": []
       }
     }
   }
   

   On the response, you should see hits and validate if it returns expected number of documents. It can show eq or gte - gte is estimation, while eq is the true number of documents
4. Now that the query is verified, you can run it by Delete by Query API. It's good idea to run the job asynchronously with wait_for_completion=false query parameter. For example:

   POST /cluster-fra-dev/_delete_by_query?wait_for_completion=false
   {
       "query": {
       "bool": {
         "must": [],
         "filter": [
           {
             "range": {
               "@timestamp": {
                 "format": "strict_date_optional_time",
                 "gte": "2023-10-26T08:45:03.732Z",
                 "lte": "2023-10-26T09:15:03.732Z"
               }
             }
           },
           {
             "match_phrase": {
               "kubernetes.namespace": "cert-manager"
             }
           }
         ],
         "should": [],
         "must_not": []
       }
     }
   }
   

5. Task is created, you can check the status by querying Task API with GET /_tasks/<task_id>.