Example - Elasticsearch template

◀

Installing Elasticsearch and Kibana with ansible

We will be using the following ansible roles available on ansible galaxy hub:

  • elastic.elasticsearch,7.4.1
  • maricaantonacci.kibana,7.4.1
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
- name: Simple Elasticsearch Example
  hosts: localhost
  roles:
    - role: elastic.elasticsearch
  vars:
    es_version: 7.4.1
    es_config:
      network.bind_host: 0.0.0.0
      discovery.type: single-node
    es_enable_xpack: true
    es_api_basic_auth_username: elastic
    es_api_basic_auth_password: 12qwas
    es_users:
      native:
        kibana:
          password: 12qwas

Warning

Take care of setting proper passwords in the playbook

1
2
3
4
5
6
7
8
- name: Simple Kibana Example
  hosts: localhost
  roles:
    - role: geerlingguy.kibana
  vars:
    kibana_version: 7.4.1
    kibana_elasticsearch_username: kibana
    kibana_elasticsearch_password: 12qwas

Modeling the topology with TOSCA

Elasticsearch SoftwareComponent node

1
2
3
4
5
6
7
8
9
    elasticsearch:
      type: tosca.nodes.indigo.Elasticsearch
      properties:
        es_version:  { get_input: es_version }
        bind_address: { get_input: es_bind_address }
        elastic_password: { get_input: es_password }
        kibana_system_password: { get_input: kibana_password }
      requirements:
        - host: server

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
  tosca.nodes.indigo.Elasticsearch:
    derived_from: tosca.nodes.SoftwareComponent
    properties:
      es_version:
        type: string
        required: false
        default: 7.4.1
      bind_address:
        type: string
        required: false
        default: 0.0.0.0
      discovery_type:
        type: string
        required: false
        default: single-node
      enable_security:
        type: boolean
        default: true
        required: false
      elastic_password:
        type: string
        required: false
        default: changeme
      kibana_system_password:
        type: string
        required: false
        default: changeme
    artifacts:
      es_role:
        file: elastic.elasticsearch,7.4.1
        type: tosca.artifacts.AnsibleGalaxy.role
    interfaces:
      Standard:
        configure:
          implementation: https://raw.githubusercontent.com/indigo-dc/tosca-types/k8s/artifacts/elk/elasticsearch_install.yml
          inputs:
            es_version: { get_property: [ SELF, es_version ] }
            bind_host: { get_property: [ SELF, bind_address ] }
            discovery_type: { get_property: [ SELF, discovery_type ] }
            enable_security: { get_property: [ SELF, enable_security ] }
            elastic_password: { get_property: [ SELF, elastic_password ] }
            kibana_system_password: { get_property: [ SELF, kibana_system_password ] }

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
---
- hosts: localhost
  connection: local
  vars:
    es_config:
      network.bind_host: "{{ bind_host }}"
      discovery.type: "{{ discovery_type }}"
    es_enable_xpack: "{{ enable_security }}"
    es_api_basic_auth_username: elastic
    es_api_basic_auth_password: "{{ elastic_password  }}"
    es_users:
      native:
        kibana:
          password: "{{ kibana_system_password }}"
  roles:
    - role: elastic.elasticsearch

Kibana SoftwareComponent node

1
2
3
4
5
6
7
8
    kibana:
      type: tosca.nodes.indigo.Kibana
      properties:
        kibana_version:  { get_input: es_version }
        elasticsearch_password: { get_input: kibana_password }
      requirements:
        - host: server
        - dependency: elasticsearch

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
  tosca.nodes.indigo.Kibana:
    derived_from: tosca.nodes.SoftwareComponent
    properties:
      kibana_version:
        type: string
        required: false
        default: 7.4.1
      elasticsearch_url:
        type: string
        required: false
        default: "http://localhost:9200"
      elasticsearch_username:
        type: string
        required: false
        default: kibana
      elasticsearch_password:
        type: string
        required: false
        default: changeme
    artifacts:
      es_role:
        file: maricaantonacci.kibana,7.4.1
        type: tosca.artifacts.AnsibleGalaxy.role
    interfaces:
      Standard:
        configure:
          implementation: https://raw.githubusercontent.com/indigo-dc/tosca-types/k8s/artifacts/elk/kibana_install.yml
          inputs:
            kibana_version: { get_property: [ SELF, kibana_version ] }
            kibana_elasticsearch_url: { get_property: [ SELF, elasticsearch_url ] }
            kibana_elasticsearch_username: { get_property: [ SELF, elasticsearch_username ] }
            kibana_elasticsearch_password: { get_property: [ SELF, elasticsearch_password ] }

1
2
3
4
5
6
---
- hosts: localhost
  connection: local
  vars:
  roles:
    - role: maricaantonacci.kibana

Now let's build the full topology template, adding the server node(s).

Don't forget to request port 5601 for accessing kibana dashboard!

1. All-in-One installation

1
Click on the "TOSCA template" tab to see a possible solution.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
tosca_definitions_version: tosca_simple_yaml_1_0

imports:
  - indigo_custom_types: https://raw.githubusercontent.com/indigo-dc/tosca-types/k8s/custom_types.yaml

description: >
  Start Elasticsearch + Kibana on a Virtual Machine

topology_template:

  inputs:
    num_cpus:
      type: integer
      description: Number of virtual cpus for the VM
      default: 2
      constraints:
      - valid_values: [ 2, 4 ]
    mem_size:
      type: scalar-unit.size
      description: Amount of memory for the VM
      default: 4 GB
      constraints:
      - valid_values: [ 4 GB, 8 GB ]

    es_version:
      type: string
      default: 7.4.1
      description: Elasticsearch version

    es_bind_address:
      type: string
      default: 0.0.0.0
      description: Bind address for Elasticsearch service

    es_password:
      type: string
      required: true
      description: Password for user elastic

    kibana_password:
      type: string
      required: true
      description: Password for kibana system user

  node_templates:

    elasticsearch:
      type: tosca.nodes.indigo.Elasticsearch
      properties:
        es_version:  { get_input: es_version }
        bind_address: { get_input: es_bind_address }
        elastic_password: { get_input: es_password }
        kibana_system_password: { get_input: kibana_password }
      requirements:
        - host: kibana_es_server

    kibana:
      type: tosca.nodes.indigo.Kibana
      properties:
        kibana_version:  { get_input: es_version }
        elasticsearch_password: { get_input: kibana_password }
      requirements:
        - host: kibana_es_server
        - dependency: elasticsearch

    kibana_es_server:
      type: tosca.nodes.indigo.Compute
      capabilities:
        endpoint:
          properties:
            network_name: PUBLIC
            ports:
              kibana:
                protocol: tcp
                source: 5601
        host:
          properties:
            num_cpus: { get_input: num_cpus }
            mem_size: { get_input: mem_size }
        os:
          properties:
            distribution: ubuntu
            type: linux
            version: 16.04

  outputs:
    kibana_endpoint:
      value: { concat: [ 'http://', get_attribute: [ kibana_es_server, public_address, 0 ], ':5601' ] }
    node_ip:
      value: { get_attribute: [ kibana_es_server, public_address, 0 ] }
    node_creds:
      value: { get_attribute: [ kibana_es_server, endpoint, credential, 0 ] }

2. Installation on separate servers

1
Click on the "TOSCA template" tab to see a possible solution.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
tosca_definitions_version: tosca_simple_yaml_1_0

imports:
  - indigo_custom_types: https://raw.githubusercontent.com/indigo-dc/tosca-types/k8s/custom_types.yaml

description: >
  Start Elasticsearch + Kibana on separate Virtual Machines

topology_template:

  inputs:
    num_cpus:
      type: integer
      description: Number of virtual cpus for the VM
      default: 2
      constraints:
      - valid_values: [ 2, 4 ]
    mem_size:
      type: scalar-unit.size
      description: Amount of memory for the VM
      default: 4 GB
      constraints:
      - valid_values: [ 4 GB, 8 GB ]

    es_version:
      type: string
      default: 7.4.1
      description: Elasticsearch version

    es_bind_address:
      type: string
      default: 0.0.0.0
      description: Bind address for Elasticsearch service

    es_password:
      type: string
      required: true
      description: Password for user elastic

    kibana_password:
      type: string
      required: true
      description: Password for kibana system user

  node_templates:

    elasticsearch:
      type: tosca.nodes.indigo.Elasticsearch
      properties:
        es_version:  { get_input: es_version }
        bind_address: { get_input: es_bind_address }
        elastic_password: { get_input: es_password }
        kibana_system_password: { get_input: kibana_password }
      requirements:
        - host: es_server

    kibana:
      type: tosca.nodes.indigo.Kibana
      properties:
        kibana_version:  { get_input: es_version }
        elasticsearch_password: { get_input: kibana_password }
        elasticsearch_url: { concat: [ 'http://', { get_attribute: [ es_server, private_address, 0 ] }, ":9200" ] }
      requirements:
        - host: kibana_server
        - dependency: elasticsearch

    es_server:
      type: tosca.nodes.indigo.Compute
      capabilities:
        host:
          properties:
            num_cpus: { get_input: num_cpus }
            mem_size: { get_input: mem_size }
        os:
          properties:
            distribution: ubuntu
            type: linux
            version: 16.04

    kibana_server:
      type: tosca.nodes.indigo.Compute
      capabilities:
        endpoint:
          properties:
            network_name: PUBLIC
            ports:
              kibana:
                protocol: tcp
                source: 5601
        host:
          properties:
            num_cpus: { get_input: num_cpus }
            mem_size: { get_input: mem_size }
        os:
          properties:
            distribution: ubuntu
            type: linux
            version: 16.04

  outputs:
    kibana_endpoint:
      value: { concat: [ 'http://', get_attribute: [ kibana_server, public_address, 0 ], ':5601' ] }
    kibana_node_ip:
      value: { get_attribute: [ kibana_server, public_address, 0 ] }
    kibana_node_creds:
      value: { get_attribute: [ kibana_server, endpoint, credential, 0 ] }

Run the deployment

Choose one of the two topologies and submit the template to the Orchestrator:

1
orchent depcreate template.yml '{ "es_password": "****", "kibana_password": "****" }'

Monitor the status of the deployment:

1
orchent depshow <dep UUID>
You could also connect to the dashboard to follow the deployment log:

dashboard URL: https://dodas-paas.cloud.ba.infn.it

What do you see in the log?

Once your deployment is finalized, you will get the outputs. E.g.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$ orchent depshow 11ea1028-4029-d1bf-a0d7-02426e42bf8c

Deployment [11ea1028-4029-d1bf-a0d7-02426e42bf8c]:
  status: CREATE_COMPLETE
  creation time: 2019-11-26T08:39+0000
  update time: 2019-11-26T08:52+0000
  callback:
  status reason:
  task: NONE
  CloudProviderName: RECAS-BARI
  outputs:
  {
      "kibana_endpoint": "http://90.147.75.103:5601",
      "kibana_node_creds": {
          "token": "-----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEAqCKFbwaYJq57fgcZQ0/xKT75J9S4GRU9 [....]\n-----END RSA PRIVATE KEY-----",
          "token_type": "private_key",
          "user": "cloudadm"
      },
      "kibana_node_ip": "90.147.75.103"
  }
  links:
    self [https://dodas-paas.cloud.ba.infn.it/orchestrator/deployments/11ea1028-4029-d1bf-a0d7-02426e42bf8c]
    resources [https://dodas-paas.cloud.ba.infn.it/orchestrator/deployments/11ea1028-4029-d1bf-a0d7-02426e42bf8c/resources]
    template [https://dodas-paas.cloud.ba.infn.it/orchestrator/deployments/11ea1028-4029-d1bf-a0d7-02426e42bf8c/template]

Verify the deployment visiting the kibana endpoint and login using the username elastic and the password set for the input es_password.