Update your skills here

Wednesday, 15 March 2017

Docker - Introduction

Docker - It's pa platform for developing, shipping and running applications using container virtualization technology

Tools as part of Docker platform:

Docker Engine
Docker Hub
Docker Machine
Docker Swarm
Docker Copmpose
Kitematic (GUI client instead CLI client)

Amazon webservices - ?
rackspace hosting - ?

Container based virtualization uses the kernel on the host's operating system to run multiple guest instances

Containers vs VMs :

Containers are more lightweight
No need to isntall guest OS
Less CPU, RAM, storage space required
More containers per machine than VMs
Greater portability

Docker Engine - It is the program that enables containers to be built, shipped and run.

Installing docker in Ubuntu:-

wget -qo- https://get.cocker.com/ | sh -> It will give all the commands in the script to shell to run , it takes care of intalling tall the dependancies.

sudo docker run hello-world -> to check whether docker installed seamlessly

to avoid using sudo before every command , add user to docker group with the following command

sudo usermod -aG docker johnnytu(user)
exit and relogin
now run docker run hello-world directly without sudo it works

this tells we have installed docker in Ubuntu

Docker daemon or Docker engine both are same.

In general we have Docker client an Docker daemon installed in the same host.
We can have Docker Client interacting with Docker Daemon installed in another host.

Client/Server architecture
Client takes user inputs and send them to the daemon
Daemon builds, runs and distributes containers
Client and daemon can run on the same host or on different hosts
CLI client and GUI(Kitematic)

sudo docker version -> to know client and server versions
Docker Hub -> It is the public registry that contains a large number of images available for your use

Docker Orchestration tools : Docker Machine, Docker Swarm and Docker Compose
Orchestration -> automated arrangements, coordination and management of complex computer systems and services

https://hub.docker.com/

docker run ubuntu:14.04 echo "Hello World"

docker images

docker run -i -t ubuntu:14.04 /bin/bash

root>adduser johnny
pwd: xxxx

root>add the user into sudo group

root>adduser johnny sudo

root>su johnny

johnny> sudo apt-gat install vim -> installing vim test editor
johnny> vim test -> this tells vim got installed and operational.

When you container process got stopped you have actually stopped container too.

exit from johnny and then from root stops the bash process that also stops container
Now, if you run docker run -i -t ubuntu:14.04 /bin/bash again, you don't see the user Johnny since it's a new container.
su johnny
cat /etc/passwd

------

docker run ubuntu echo "hello world" ->container has started echoed and then stopped

docker run -it ubuntu bash
root> ps -ef

Now bash has pid 1 inside the container

Now exit the container without stopping the container by pressing ctrl+P+Q

now check the processes with ps -ef

----

docker ps -> lists the containers up and running
docker ps -a -> lists the containers up and running along with stopped already

----
container running in detached mode i.e in the background
docker run -d centos:7 ping 127.0.0.1 -c 100

check container is running with
docker ps

docker logs <container id> -> gives 100times ping command - it's a way to inspect the output of the that container
docker logs -f <container id> -> it's attached to log file, similar to tail -f

--------------
download tomcat

docker run -d -P tomcat:7
it's up and running now, you can check that with docker ps command

------------ part2

commit docker images

docker run -it ubuntu:14.04 bash -> opens bash
root>curl 127.0.0.1
install curl with
root>apt-get install -y curl

exit bash means container

docker ps -a

docker commit <container-id> johnnytu/myapp:1.0

docker images
docker run -it johnnytu/myapp:1.0 bash
root> which curl -> check curl is installed
root> curl 127.0.0.1

Dockerfile : It's a configuration file that contains instructions for building a Docker image.

--------
build an image from docker file

mkdir test
cd test
vim Dockerfile

FROM UBUNTU:14:04
RUN apt-get update
RUN apt-get unstall -y curl
RUN apt-get install -y vim

save and exit

test> docker build -t johnnytu/testimage:1.0 .

docker images -> to verify image created

vim Dockerfile

FROM UBUNTU:14:04
RUN apt-get update && apt-get install -y curl \ vim

save and exit

docker build -t jonnytu/testimage:1.0 .

vim Dockerfile

FROM UBUNTU:14:04
RUN apt-get update && apt-get install -y curl \ vim
CMD ["ping","127.0.0.1","-c","30"] ->it pings 30 times

save and exit
test>docker build -t johnnytu/testimage:1.1 .
docker run johnnytu/testimage:1.1 ->to run the new image
docker run johnnytu/testimage:1.1 echo "hello world" -> override CMD instruction

-----
ENTRYPOINT

im Dockerfile

FROM UBUNTU:14:04
RUN apt-get update && apt-get install -y curl \ vim
ENTRYPOING ["ping"]

save and exit

test>docker build -t johnnytu/testimage:1.2 .
docker run johnnytu/testimage:1.2
docker run johnnytu/testimage:1.2 127.0.0.1 -c 5

-----
container start/stop

docker run -d nginx ->run in background
docker ps
docker stop <container-id>
docker ps -> now you can see there is no active containers
docker ps -a
docker start condencending_jang<container-name>
docker ps -> now you can check container is running

--------
continer exec

docker run -d tomcat:7
docker ps
docker exec -it <container-id> bash
root>/usr/local/tomcat/>
root> ps -ef
-------
docker rm

docker ps
docker stop lonely_goldstine
docker ps -a
docker rm lonely_goldstine
docker ps -a
----------
docker rmi

docker images
docker rmi johnnytu/testimage:1.1
docker images

docker rmi <image-id>

--------
docker push

docker push johnnytu/restimage:1.0
loging to repo to push with command interactive mode

it should fail because in repo it didn't find johnnytu/restimage:1.0 , to avoid it you should tag the image first.

docker tag johnnytu/testimage:1.0 trainingteam/testimage:1.0
docker images

docker push trainingteam/testimage:1.0 ->it's sending image to docker hub

-----------

Volumes - A Volume is a designated directory ina container, which is designed to persist data, independent of the container's life cycle.

docker run -d -P -v /www/website nginx
docker exec -it <container-id> bash
root> cd www/website
root>/www/website/> echo "hell" >>test
root>/www/website/> ls -> test
cat test -> hello
exit
docker stop <container-id>
docker commit <container-id> test:1.0
docker run -it test:1.0 bash
root> ls -> www\website -> ls --> there is no file
Volume data has been excluded when updating an image

----------
ports

docker run -d -p 8080:80 nginx
docker ps
----------
docker linking

docker run -d --name dbms postgres
docker ps
docker run -it --name website --link dbms:db ubuntu:14.04 bash
root> cat /etc/hosts
an entry will be created with db in hosts file with ip address of dbms container

check this with following command
docker inspect dbms | grep IPAddress -> it displays ip address

----------------
Docker in contentious Integration

In CI you can compile and run as part of Image

Docker Maven PlugIn:


<dependency>
<groupId>com.spotify</groupId>
<artifactId>docker-maven-plugin</artifactId>
<version>0.4.13</version>

</dependency>

Home Page : https://github.com/spotify/docker-maven-plugin

https://github.com/VamshiKrishnaNemalikonda/deegeu-maven-java-docker-example

Create Docker Container:
mvn clean package docker:build (Used in development when you want to create a Docker container locally)

Create/Push Docker Container:

mvn clean package docker:build -DpushImage (Used in continuous integration tools when you want to deploy the resulting Docker container to Docker Hub or a private repository)

Tuesday, 14 March 2017

DevOps - Introduction

Wednesday, 17 August 2016

Adding multiple jars to dependency

<dependency>
   <groupId>foo</groupId>
   <artifactId>foo</artifactId>
   <version>1.0</version>
   <scope>system</scope>
   <systemPath>${basedir}/lib/*.jar</systemPath>
  </dependency>

<plugin>
     <groupId>com.googlecode.addjars-maven-plugin</groupId>
     <artifactId>addjars-maven-plugin</artifactId>
     <version>1.0.2</version>
     <executions>
      <execution>
       <goals>
        <goal>add-jars</goal>
       </goals>
       <configuration>
        <resources>
         <resource>
          <directory>${basedir}/../buildtools/lib</directory>
         </resource>
        </resources>
       </configuration>
      </execution>
     </executions>
    </plugin>

Wednesday, 10 August 2016

Listing of the elements directly under the POM's project element

Saturday, 6 August 2016

Elasticsearch

===================================================Kibana==================================================
===========================================================================================================
Kibana is an open source analytics and visualization platform designed to work with Elasticsearch.
You use Kibana to search, view, and interact with data stored in Elasticsearch indices.
You can easily perform advanced data analysis and visualize your data in a variety of charts, tables, and maps.
Kibana makes it easy to understand large volumes of data. Its simple, browser-based interface enables you to quickly create and share dynamic dashboards that display changes to Elasticsearch queries in real time.

kibana is a reporting tool

Discover :
========
https://www.elastic.co/guide/en/kibana/current/discover.html#discover
You can interactively explore your data from the Discover page.
You have access to every document in every index that matches the selected index pattern.
You can also see the number of documents that match the search query and get field value statistics.
If a time field is configured for the selected index pattern, the distribution of documents over time is displayed in a histogram at the top of the page.

indices :
=======
C:\Users\vnemalik\Documents\001096043\soft\elasticsearch-2.1.1\bin>elasticsearch.bat
[2016-02-10 15:11:58,272][WARN ][bootstrap                ] unable to install syscall filter: syscall filtering not supported for OS: 'Windows 7'
[2016-02-10 15:11:59,024][INFO ][node                     ] [node-1] version[2.1.1], pid[6112], build[40e2c53/2015-12-15T13:05:55Z]
[2016-02-10 15:11:59,025][INFO ][node                     ] [node-1] initializing ...
[2016-02-10 15:11:59,135][INFO ][plugins                  ] [node-1] loaded [], sites []
[2016-02-10 15:11:59,247][INFO ][env                      ] [node-1] using [1] data paths, mounts [[OS (C:)]], net usable_space [245gb], net total_space [297.7gb], spins? [unknown], types [NTFS]
[2016-02-10 15:12:04,088][INFO ][node                     ] [node-1] initialized
[2016-02-10 15:12:04,088][INFO ][node                     ] [node-1] starting ...
[2016-02-10 15:12:04,281][INFO ][transport                ] [node-1] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2016-02-10 15:12:04,313][INFO ][discovery                ] [node-1] xyz/jg-IhVS2Qx-c5dN8ge9VBg
[2016-02-10 15:12:08,349][INFO ][cluster.service          ] [node-1] new_master {node-1}{jg-IhVS2Qx-c5dN8ge9VBg}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-02-10 15:12:08,376][INFO ][http                     ] [node-1] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2016-02-10 15:12:08,376][INFO ][node                     ] [node-1] started
[2016-02-10 15:12:08,756][INFO ][gateway                  ] [node-1] recovered [6] indices into cluster_state

Time Filter :
===========
The Time Filter restricts the search results to a specific time period.
Searching ( Elasticsearch Query DSL/Lucene query syntax ) :
=========================================================
status:200
status:[400 TO 499] - Lucene query syntax
status:[400 TO 499] AND (extension:php OR extension:html) - Lucene query syntax
Automatically Refreshing the Page / Refresh Interval :
====================================================
You can configure a refresh interval to automatically refresh the page with the latest index data. This periodically resubmits the search query.
Filtering By Field :
==================
You can filter the search results to display only those documents that contain a particular value in a field.
To add a positive filter, click the Positive Filter button Positive Filter Button. This filters out documents that don’t contain that value in the field.
To add a negative filter, click the Negative Filter button Negative Filter Button. This excludes documents that contain that value in the field.
Viewing Document Data :
=====================
When you submit a search query, the 500 most recent documents that match the query are listed in the Documents table.
Kibana reads the document data from Elasticsearch and displays the document fields in a table. The table contains a row for each field that contains the name of the field, add filter buttons, and the field value.
meta-fields :
===========
meta-fields include the document’s _index, _type, _id, and _source fields.

Creating Indices:
================
Creating indices using logstash
Creating indeces using Curl
curl -XPUT http://localhost:9200/twitter5
curl -XPUT 'http://localhost:9200/twitter10/' -d '{
"settings" : {
        "index" : {
            "number_of_shards" : 3,
            "number_of_replicas" : 2
        }
    }
}'

The create index API
--------------------
curl -XPUT 'http://localhost:9200/twitter10/' -d '{ "settings" : { "index" : { "number_of_shards" : 3, "number_of_replicas" : 2 } } }'

The create index API allows to provide a set of one or more mappings:
---------------------------------------------------------------------
curl -XPOST localhost:9200/test -d '{ "settings" : { "number_of_shards" : 1 }, "mappings" : { "type1" : { "_source" : { "enabled" : false }, "properties" : { "field1" : { "type" : "string", "index" : "not_analyzed" } } } } }'

curl -XPUT localhost:9200/test -d '{ "creation_date" : 1407751337000 }'
curl -XDELETE 'http://localhost:9200/twitter/'
curl -XGET 'http://localhost:9200/twitter/'
The get index API can also be applied to more than one index, or on all indices by using _all or * as index.
curl -XGET 'http://localhost:9200/twitter/_settings,_mappings'   (_settings, _mappings, _warmers and _aliase
Does )

Does Index exist:
curl -XHEAD -i 'http://localhost:9200/twitter'
Closing/Opening indexes :
curl -XPOST 'localhost:9200/my_index/_close'
curl -XPOST 'localhost:9200/my_index/_open'

PUT Mapping:
===========
1) Creates an index called twitter with the message field in the tweet mapping type.
curl -XPUT http://localhost:9200/twitter11 { "mappings": { "tweet": { "properties": { "message": { "type": "string" } } } } }
2) Uses the PUT mapping API to add a new mapping type called user.
curl -XPUT http://localhost:9200/twitter11/_mapping/user { "properties": { "name": { "type": "string" } } } - Not working
3) Uses the PUT mapping API to add a new field called user_name to the tweet mapping type.
curl -XPUT http://localhost:9200/twitter11/_mapping/tweet11 { "properties": { "user_name": { "type": "string" } } }

Kibana Search
=============
SubmitterId = "BS321GRACEZI" OR TransactionID = "8900145433765010"
"Transaction ID = 8900145433765010" AND "SUBMITTER ID = BS321GRACEZI"
LogLevel:DEBUG AND JavaClass:EDIEligibilityBO
"Transaction ID: 8900145433765010" AND "SUBMITTER ID: BS321GRACEZI" AND "B2B Error Code: 0"
TransactionID = [ 8220143989361570 TO 8900145433765010 ] AND "Submitter ID = BS321GRACEZI"
TransactionID = [ 8220143989361570 TO 8900145433765010 ] AND ("Submitter ID = BS321GRACEZI" OR "Submitter ID = B00099999800")
NOT "Submitter ID = BS321GRACEZI"
====================================================
Logstash
====================================================
bin/logstash -e 'input { stdin { } } output { stdout {} }'
https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html    - Show Logstash design
input {
    file {
        path => "/path/to/logstash-tutorial.log"
        start_position => beginning
    }
}
The default behavior of the file input plugin is to monitor a file for new information, in a manner similar to the UNIX tail -f command. To change this default behavior and process the entire file, we need to specify the position where Logstash starts processing the file.
To verify your configuration, run the following command:
bin/logstash -f first-pipeline.conf --configtest
curl -XGET http://localhost:9200/logstash-2016.02.10/_search?q=response=200
nput {
file {
    path => "/var/log/messages"
    type => "syslog"
}
file {
    path => "/var/log/apache/access.log"
    type => "apache"
}
}
path => [ "/var/log/messages", "/var/log/*.log" ]
path => "/data/mysql/mysql.log"
output {
file {
    path => "/var/log/%{type}.%{+yyyy.MM.dd.HH}"
}
}
input {
file {
    path => "/tmp/*_log"
}
}
http://localhost:9200/twitter/_settings/_index/
http://localhost:9200/logstash-*/_settings/_index
http://localhost:9200/logstash-2016.02.10/_settings/
match => { "message" => "%{COMBINEDAPACHELOG}"}
match => { "message" => "google"}
match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }
geoip {
        source => "clientip"
    }

==============================================================================================================================================================
         Elastic Search
==============================================================================================================================================================
To see all the mappings related to each index
---------------------------------------------
"@timestamp":{"type":"date","format":"dateOptionalTime"}

if [type] == "b2b_field_mapping" { }   -??

indexing, searching, and modifying your data.
There are a few concepts that are core to Elasticsearch. Understanding these concepts from the outset will tremendously help ease the learning process.
Near Realtime (NRT) :
===================
Elasticsearch is a near real time search platform. What this means is there is a slight latency (normally one second) from the time you index a document until the time it becomes searchable.

Cluster:
=======
A cluster is a collection of one or more nodes (servers) that together holds your entire data and provides federated indexing and search capabilities across all nodes. A cluster is identified by a unique name which by default is "elasticsearch". This name is important because a node can only be part of a cluster if the node is set up to join the cluster by its name.
Node:
=====
A node is a single server that is part of your cluster, stores your data, and participates in the cluster’s indexing and search capabilities. Just like a cluster, a node is identified by a name which by default is a random Marvel character name that is assigned to the node at startup.
Index:
======
An index is a collection of documents that have somewhat similar characteristics. For example, you can have an index for customer data, another index for a product catalog, and yet another index for order data. An index is identified by a name (that must be all lowercase) and this name is used to refer to the index when performing indexing, search, update, and delete operations against the documents in it.

Within an index/type, you can store as many documents as you want. Note that although a document physically resides in an index, a document actually must be indexed/assigned to a type inside an index.

Used to check if the index (indices) exists or not.
curl -XHEAD -i 'http://localhost:9200/twitter'

Points to Note:
==============

logstash -f b2bLog.conf --log C:/KibanaElasticSearch/StageVersion/logstash-2.3.2/logstash.log &

Elasticsearch is hosted on Maven Central. ( http://search.maven.org/#search|ga|1|a%3A%22elasticsearch%22 )
//grok condition
if '"B2B Error Code"' not in [kvpairs] {
            json {
                source => "kvpairs"
                remove_field => [ "kvpairs" ]
                add_field => {"Transaction_Status" => "UNSUCCESSFUL,Error Code Not Found"}
            }
        }
// Grok match string for b2b log
match => { "message" => "\[%{LOGLEVEL:LogLevel}\] %{MONTHDAY:Date} %{MONTH:Month} %{YEAR:Year} %{TIME:Timestamp} - %{DATA:JavaClass} %{DATA:JavaMethod}- %{GREEDYDATA:CorrelationID}: %{GREEDYDATA:kvpairs}"}
match => { "message" => "\[%{LOGLEVEL:LogLevel}\] %{B2B_DATE:timestamp} - %{DATA:JavaClass} %{DATA:JavaMethod}- %{GREEDYDATA:CorrelationID}: %{GREEDYDATA:kvpairs}"}
// To list out all the mapping associated to each index - GET operationhttp://localhost:9200/_all/_mapping?pretty=1
// To list out the single template - GET operationhttp://localhost:9200/_template/logstash?pretty
// To list all the templates availablehttp://localhost:9200/_template/

Elastic Search DSL(Domain Specific Language).
index => "logstash-gpuz-%{+YYYY.MM.dd}"
"format": "yyyy-MM-dd HH:mm:ss"
manage_template => false if you want to manage the template outside of logstash.
Disable the option Use event times to create index names and put the index name instead of the pattern (tests).
Default for number_of_replicas is 1 (ie one replica for each primary shard)
curl -XGET 'http://localhost:9200/twitter/_settings,_mappings' - get api for index
The above command will only return the settings and mappings for the index called twitter.
The available features are _settings, _mappings, _warmers and _aliases.
1)Installing sense plug-in for kibana
kibana.bat plugin --install elastic/sense
2) another way, download and add plugin
https://download.elasticsearch.org/elastic/sense/sense-latest.tar.gz
https://download.elastic.co/elastic/sense/sense-latest.tar.gz - latest
$ bin/kibana plugin -i sense -u file:///PATH_TO_SENSE_TAR_FILE
https://www.elastic.co/guide/en/sense/current/installing.html
two ways to ovveride the existing logstash template:
1) manage_template => true
template_overwrite => true
template_name => "b2btemplate"
template => "C:/Users/vnemalik/Documents/001096043/soft/logstash-2.1.1/templates/automap.json"
{
    "template": "logstash-*",

    "settings": {
"number_of_shards" : 1
    },
    "mappings": {
        "b2bkibana": {
            "_all": {
                "enabled": true
            },
            "properties": {
                "@timestamp": {
                    "type": "date",
                    "format": "dateOptionalTime"

            },
  "@version": {
                    "type": "string"

                },

                "CorrelationID": {
                    "type": "string",
                    "index": "not_analyzed"
                },
   "Submitter ID": {
                    "type": "string",
                    "index": "not_analyzed"
                },

                "Transaction Type": {
                    "type": "long",
                    "index": "not_analyzed"
                },

                "Transaction Version": {
                    "type": "string",
                    "index": "not_analyzed"
                },
  "Transaction Mode": {
                    "type": "string",
                    "index": "not_analyzed"
                },

                "Transaction ID": {
                    "type": "long",
                    "index": "not_analyzed"
                },

                "ServiceTypeCode": {
                    "type": "long",
                    "index": "not_analyzed"
                },

                "Payer ID": {
                    "type": "long",
                    "index": "not_analyzed"
                },
  "Service invoked": {
                    "type": "string",
                    "index": "not_analyzed"
                },

                "Service type": {
                    "type": "string",
                    "index": "not_analyzed"
                },

                "<statusMessageLevel>": {
                    "type": "string",
                    "index": "not_analyzed"
                },
  "<serviceCallStatus>": {
                    "type": "string",
                    "index": "not_analyzed"
                },

                "<messageType>": {
                    "type": "string",
                    "index": "not_analyzed"
                },

                "<statusMessage>": {
                    "type": "string",
                    "index": "not_analyzed"
                },

                "System ID": {
                    "type": "string",
                    "index": "not_analyzed"
                },

                "Source Code for Coverage": {
                    "type": "string",
                    "index": "not_analyzed"
                },

                "Claim System Type Code for Coverage": {
                    "type": "string",
                    "index": "not_analyzed"
                },

                "Eligibility System Type Code for Coverage": {
                    "type": "string",
                    "index": "not_analyzed"
                },

                "Coverage Type": {
                    "type": "string",
                    "index": "not_analyzed"
                },

                "Vendored Coverage": {
                    "type": "string",
                    "index": "not_analyzed"
                },

                "Vendor Name": {
                    "type": "string",
                    "index": "not_analyzed"
                },

                "Source Code": {
                    "type": "string",
                    "index": "not_analyzed"
                },

                "Claims System Type Code": {
                    "type": "string",
                    "index": "not_analyzed"
                },

                "Eligiblity System Type Code": {
                    "type": "string",
                    "index": "not_analyzed"
                },

                "B2B Error Code": {
                    "type": "string",
                    "index": "not_analyzed"
                },

                "AAA03": {
                    "type": "string",
                    "index": "not_analyzed"
                },

                "AAA04": {
                    "type": "string",
                    "index": "not_analyzed"
                },

  "JavaMethod": {
                    "type": "string",
                    "index": "not_analyzed"
                },
                "JavaClass": {
                   "type": "string",
                   "index": "not_analyzed"
                },
                "LogLevel": {
                    "type": "string",
                    "index": "not_analyzed"
                },
                "Date": {
                    "type": "date",
                    "index": "not_analyzed"
                }

        }
    }
}
}
// Output plug in to skip grok failures
output {
if [type] == "apache-access" {
    if "_grokparsefailure" in [tags] {
      null {}
    }

    elasticsearch {
    }
}
}
2) Either from curl or Fiddler Web Debugger or Sense tab of Kibana
edidashboardtemplate

installing aggrigate plugin:
C:\Users\vnemalik\Documents\001096043\b2b\ElasticSearch_POC\testing_nodes\logstash-2.1.1\bin>plugin install logstash-filter-aggregate

// Elastic boxsudo su -c "sh elasticsearch" -s /bin/sh aneela1
sudo -b su -c "sh elasticsearch" -s /bin/sh aneela1
sudo -b su -c "sh kibana" -s /bin/sh aneela1
Start - $nohup bin/kibana &
Stop – kill -9 (pid)
now-1w/w
apsrs3723 - Initial Stage Dashboard
apsrs3726 - thats the latest version and also have NAS connected to it - its the stage server in the DMZ!
we have our stage servers(apsp8705,apsp9016,) connected to apsrs3926(linex server) via NAS share..
From apsp8705(AIX) logs(x12logs, processlogs) shipped to apsrs3926(linux,/b2b_lt/elastic) where ES stack got installed.

Useful Links:
============

https://www.youtube.com/watch?v=60UsHHsKyN4
https://www.youtube.com/watch?v=U3m0jKygAqU
http://code972.com/blog/2015/02/80-elasticsearch-one-tip-a-day-managing-index-mappings-like-a-pro
https://www.timroes.de/
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html
https://discuss.elastic.co/t/cannot-get-my-template-to-work/27150/15    -good one , templates https://discuss.elastic.co/t/confused-about-how-to-use-raw-fields-and-not-analyze-string-fields/28106
http://edgeofsanity.net/article/2012/12/26/elasticsearch-for-logging.html
http://cookbook.logstash.net/recipes/cisco-asa/ -?

Do you want this? The mutate filter will change all the double quote to single quote.
filter {
    mutate {
        gsub => ["message","\"","'"]
    }
}
mutate {
        gsub => ['message','\"','`']
    }
match => { "message" => "(?m)\[%{LOGLEVEL:LogLevel}\] %{B2B_DATE:editimestamp} - %{DATA:JavaClass} %{DATA:JavaMethod}- %{GREEDYDATA:CorrelationID}: %{GREEDYDATA:kvpairs}"}
timestamp issue
----------------https://discuss.elastic.co/t/how-to-set-timestamp-timezone/28401/16