Questions tagged [logstash]

Logstash is a tool for managing events and logs. You can use it to collect logs, parse them and send them to storage for later use (such as searching).

0
votes
1answer
8 views

How to grok catalina log file

I'm trying to find a pattern for this line of log (extracted from catalina.log) of an apache tomcat 8 installation. 30-Apr-2019 15:40:40.044 INFOS [main] org.apache.catalina.startup....
0
votes
0answers
20 views

Logstash - Solr Integration

Below is my actual log, --2019-05-09 06:49:05.590 -TRACE 6293 --- [ntainer#0-0-C-1] c.s.s.service.MessageLogServiceImpl : [...
0
votes
0answers
13 views

Logstash grok patterns special characters

I have a problem in my grok pattern with special characters. I use filebeat to send my logs in logstash. And in logstash I use grok patterns to parse elements. My logs : 5/19/2019 7:27:32 PM | APPLI=...
0
votes
1answer
15 views

i want to change dashboard background color to Black in kibana 7.0.1 version

I am using the Kibana 7.0.1 and I want to change the background color of the complete dashboard to black color, but I cannot find this option on UI. Is there any way to do this ? Thanks!
0
votes
0answers
16 views

logstash grok issue while filtering data

I have a data that's basically for data deletion via rm command which looks as follows. ttmv516,19/05/21,03:59,00-mins,dvcm,dvcm 166820 4.1 0.0 4212 736 ? DN 03:59 0:01 rm -rf /dv/project/agile/...
0
votes
1answer
18 views

Logstash ignoring multiple date filters

i'm making a logstash .conf, and on my filter i need to extract the weekday of two timestamps, but Logstash act as if he only is making one match, example: Timestamp 1: Mar 7, 2019 @ 23:41:40.476 . ...
0
votes
1answer
11 views

Logstash config output index daily by date provided by data column

My data has a column that has a recorded_date on it with the format yyyy-MM-dd HH:mm:ss. I would like to index my data daily, but I'm not sure how to append a different date format on my index name, ...
0
votes
0answers
10 views

Custom Domain in Logstash S3 Output Plugin

I'm running into a wall trying to figure out how to get rid of ".amazonaws.com" from the URL that is built from the Logstash S3 output plugin. It does not work in the environment I am in. I have tried ...
0
votes
0answers
13 views

How to get the week number of the month in logstash

everyone, i need to add a field which contains a week number, using logstash filters. I read all documentation of the date filter and i try some whit date format and date filter, i try it whit this ...
0
votes
0answers
10 views

Logstash - Multiple grok pattern not working together

I am very new in using Logstash. I have two kinds of log, Pattern 1 : --2019-05-09 08:53:45.057 -INFO 11736 --- [ntainer#1-0-C-1] c.s.s.service.MessageLogServiceImpl : [...
0
votes
0answers
12 views

Ingest data from encrypted sqlite (sqlcipher) with logstash

I want to ingest data from a sqlite file that is encrypted with sqlcipher. but there is not any option in default jdbc driver for my purpose. I found this fork of driver but it's not working either: ...
0
votes
0answers
15 views

Unable to retain client-IP when using LoadBalancer service instead of pod

We're setting up an ELK stack on Kubernetes, utilizing the opendistro version of Elasticsearch and Kibana. We are also using the OSS version of Logstash to ingress logs. In front of the ELK stack/...
0
votes
1answer
22 views

Logstash/Gelf not stored all MDCs

I'm using Logstash/Gelf as a log-tools in my application. I follow this sample for setting up my logger as JSON file. here is my wildfly config (standalone.xml): <profile> <subsystem ...
0
votes
0answers
17 views

How to push Hive query result to elasticsearch using logstash

Got multiple Hive tables, each having 1M+ records. Run fairly complicated queries by joining multiple tables. Would like to push the result to an elastic stack using Logstash's input plugin. ...
0
votes
0answers
18 views

Solution to bypass Logstash mixing logs

I'm currently in front of a structure probleme with logstash. I have a syslog-ng client sending logs from different files through the network to an ELK stack. I noticed that Logstash is mixing logs, ...
-1
votes
0answers
12 views

logstash input jdbc plugin monetdb not working

Monetdb does not attempt to connect at all. MySQL and MSSQL, connection attempts are performed normally. input { jdbc { jdbc_driver_library => "/etc/logstash/monetdb-jdbc-2.28.jar" ...
0
votes
1answer
35 views

How to get around “Detected ambiguous Field Reference” when parsing HTTP URL parameters with Logstash

So I have been using Logstash to parse Apache access logs. Currently I am using some kv filters to handle this with Logstash 6.7.x (and earlier) with but wanted to upgrade to Logstash 7.x. It seems ...
-1
votes
1answer
16 views

Logstash how to split a field in logstash

i want help in split the field below, I want to create array of all the occurrence in the field below which starts with "START - PHASE 1 09:02:16.31, time is the dynamic value generated. Please ...
0
votes
1answer
29 views

How to parse json in logstash /grok from ansible_results

I have below messages from ansible_results which i'm trying to parse, what basically i need is to cut the down the field after "msg": from the below messages. Log sample: 2019-05-07 07:56:06,374 p=...
1
vote
0answers
21 views

Create URL on string fields in kibana using two filters , the value of the field and the timestamp

I am looking to create URL on string field called "session name" in kibana interface to be redirected to an other search. In my URL I need to use two queries, one based on the field value, for that ...
0
votes
2answers
21 views

How to import data from elasticsearch to sql server database?

I am trying to import data from elasticsearch to sql server database, I have done import of data from sqlserver to elastic search using logstash, I reversed input and output but its throwing the error ...
0
votes
0answers
20 views

Exporting DB data to ElasticSearch

What are the available options to keep Oracle DB data in sync with ElasticSearch v7.0 indices? The DB is Oracle DB. The DB data would include multiple Business Objects each of which is the result of ...
0
votes
1answer
21 views

Logstash Error - Unknown setting '“hosts”' for elasticsearch

I am trying to load data from oracle to elastic using logstash. Elastic 7.0.1 Logstash 7.0.1 logstash.conf: input { jdbc { # The path to our downloaded jdbc driver ...
0
votes
1answer
30 views

How can I retrieve the tag from the syslog logs than are sent to Logstash?

I have set up my Docker daemon so that the logs of all my containers are forwarded to a Logstash application listening on port 5000, using the following configuration for daemon.json : { "log-...
0
votes
1answer
22 views

Remove characters from JSON

I try to parse some json with logstash, currently the file which I like to enter has the following structure (simplified): -4: {"audit":{"key1":"value1","key2":"value2"}} -4: {"audit":{"key1":"value1"...
2
votes
1answer
31 views

Logstash Filter, add field from split array, if not empty

I am doing a split on two fields, and assigning different array elements to new fields. However when they dont exist it ends up assinging the code to the field, e.g"%{variable}" I assume I could do ...
0
votes
0answers
17 views

Can i apply pipeline.id in filter grok pattern by putting IF statement.. to get the exact pattern based on pipeline ID

if [type] == "" { grok { match => { "message" => ["%{LOGLEVEL:log} \[%{DATA:class}\] %{NOTSPACE:Property} %{SPACE} %{NOTSPACE} %{NOTSPACE} (?<id>[^ ]*) %{NOTSPACE} %{NOTSPACE} %{...
0
votes
0answers
15 views

Logstash 2.3.4 How to load nested document in elasticsearch using logstash-jdbc plugin

I am currently using elasticsearch 2.3.4 and logstash 2.3.4 to load relational data from Oracle db into my elasticsearch index using logstash-jdbc plugin. As suggested in various posts, I am using ...
0
votes
1answer
48 views

Failed to execute action :action=>LogStash::PipelineAction::Create/pipeline_id:main

I have installed ELK stack version 7.0.0 on my CentOS7 VM and I faced with an issue during Logstash service start: [ERROR] 2019-05-13 08:21:37.359 [Converge PipelineAction::Create] agent - Failed to ...
0
votes
1answer
76 views

Import a large sql file (30 gb per day) to one single index in elasticseach

I have sql dump (40 GB) generated every day, i would like to migrate this dump to elasticseach in one single index ex: the dump of yesterday was transferred to index1, the dump of today should be ...
0
votes
1answer
22 views

Block unauthorized and distinguish users

Logstash has an open port where everyone can send in data. Anonymous data messes everything up All data from all customers in one pool is messed up too. So I read and tried https://www.elastic.co/...
0
votes
0answers
23 views

How to find the total number of request and response of a server?

i need find the total number of request and response count of a server.So i need to know
0
votes
0answers
33 views

Spring boot microservice not pushing logs to logstash

I have a microservice which is not pushing logs to logstash For Centralized logging i am using ELK Stack, which is running in Docker Container I am using logback.xml in my spring boot code <?xml ...
0
votes
0answers
17 views

Logstash tomcat stack trace parsing failed

I'm trying to parse some Tomcat logs with logstash, here is an example of data : 07-05-19 11:24:32 INFO [XmlWebApplicationContext] Closing Root WebApplicationContext: startup date [Wed Apr 10 15:58:...
0
votes
1answer
23 views

wanted to know i have a log with 2 different grok pattern… i wanted to put that in IF statement

i have 2 patterns in one log and i wanted to put that in the grok pattern with if statement 1.%{LOGLEVEL:log} \[%{DATA:class}\] %{NOTSPACE:Property} %{SPACE} %{NOTSPACE} %{NOTSPACE} (?<id>[^ ]*)...
0
votes
1answer
29 views

How to install logstash plugin with docker-compose?

I try to install logstash with a docker-compose but docker exited with code 0 just after Installation successful when I try to install a logstash plugin. The part of docker-compose file for logstash ...
1
vote
1answer
39 views

Bulk insertion of data to elasticsearch via logstash with scala

I need to insert large bulk data to elasticsearch regulary via scala code. When googling, I found to use logstash for large insertion rate but logstash doesn't have any java libraries or Api to call ...
0
votes
2answers
20 views

Clone a logstash event N number of times with the value from the message itself

How can I clone a event "N" number of times, where the value of N comes from the event itself. For eg. Input event is like this : "Event1 [Host] Machine A (4 licenses)" "Event2 [Host] Machine A (2 ...
2
votes
1answer
23 views

Illegal unquoted character for Logstash's Google Cloud Storage input plugin

I am trying to download stackdriver logs from my GCP bucket using logstash with this configuration: google_cloud_storage { interval => 60 bucket_id => "stackdriverlog\-cloudraid" ...
0
votes
0answers
17 views

Logspout SYSLOG_DATA for Docker containers

I am using Logspout "gliderlabs/logspout:latest" to route docker logs to logstash server. But i am not able to get docker container names labels in response. Below is the attached Log snapshot of ...
2
votes
3answers
57 views

How to fix duplicated documents in Elasticsearch when indexing by Logstash?

I'm using the Elastic Stack to handle my log files but is generating duplicated documents in the Elasticsearch. I've made some survey and already tried to add the "document_id", but it did not solve. ...
0
votes
0answers
16 views

Failed to sent event to MongoDB using Logstash caused by wrong argument number

I am trying to push some logs to my local MongoDB instance using Logstash with the configuration as follow: output{ mongodb { id => "mongo-aws-cloudtrail" collection => "...
0
votes
2answers
31 views

Is there a way of sending data directly to elasticsearch without using filebeats and logstash?

Till now i have sent my data to Elasticsearch using either Filebeat or Logstash and sometimes both. I just want to know whether there is any way of sending my data directly to Elasticsearch without ...
1
vote
1answer
21 views

how to use elapsed filter- logstash

I am working in the Elapsed filter. I read the guide of Elapsed filter in logstash. then i made a sample config file and csv to test the working of Elapsed filter. But it seems to be not working. ...
0
votes
1answer
44 views

How to filter java logs using logstash? [closed]

I need to filter only java fatal logs and put that logs into elasticsearch index using logstash pipeline. I put logs to an elasticsearch index. but it has all the logs(ex- info,debug). Sample log ...
-1
votes
0answers
32 views

Parsing through grok filter

These are some of the log samples shared- I have to extract fields out the following logs printed. Fields description is stated below- 2019-04-25 12:38:58|[2.1.250 - A16DOHI3 - 306ed8cf-ea45-4853-...
0
votes
2answers
23 views

How to move part of the string after exact word to another field in logstash?

Let's imagine I have log file like the following: My custom exception ST: java.lang.RuntimeException: Text of this dummy err. My final goal is to put everything after ST: to new field ST called and ...
0
votes
0answers
14 views

Unable to connect to SQL Server 2008 R2 on Windows 7 Logstash

I am running logstash 7.0 on Windows 7 with below configuration SQL Server 2008 R2 (Remote) jdbc verson 6.0. I have also used jdbc 4.0 same error is received. java version "1.8.0_211" Java(TM) SE ...
0
votes
0answers
16 views

logstash json input {}, “” filter

I want to get a json-style event that contains "" or {}. However, if "" and {} are included, filtering is not done properly. Is there no way?? msg: {"key1":"val1","key2":"val2", "key3": "asfqwe12\"...
0
votes
0answers
15 views

Query don'ts apply in logstash

everybody, when we are implementing our ELK solution we had a problem, some servers are windows 2003 because of that, we need to develop a new "beat" we create a beat in python, we maked somo mistakes,...