Lab-51: Azure IoT hub, routes and endpoints

This lab is the continuation of  Lab-50  In Lab-50 we learned how to setup Azure IoT hub, simulate  IoT device and read device to cloud messages using web app and device explorer/iothub-explorer. In this lab we will learn how to read D2C and send C2D messages programmatically using Visual Studio 2017

Before we proceed it will be good to understand few concepts in IoT hub, mainly Endpoints and Routes

azure_iot_19

Endpoints:

There are two type of endpoints in IoT hub 1) Built-in endpoints and 2) custom end-points. Think of endpoints as topics or queue in AMQP, MQTT protocols.

azure_iot_38

  1. Built-in Endpoints

For each device in the identity registry, IoT Hub exposes a set of built-in endpoints, these endpoints can  be device facing or service facing

Device side endpoints

When device sending message to cloud it uses this endpoint. If you are using Azure client SDK you don’t need to worry about setting this endpoint, SDK does it for you.

/devices/{device_id}/messages/events

Device receives C2D messages on this endpoint

/device/{device_id}/messages/devicebound

Service endpoints

Each IoT hub exposes a set of endpoints for your solution back end to communicate with your devices. With one exception, these endpoints are only exposed using the AMQP protocol.

D2C endpoint. This endpoint is compatible with Azure Event Hubs. A back-end service can use it to read device-to-cloud messages sent by your devices.

/messages/events

C2D endpoints enable your solution back end to send reliable cloud-to-device messages, and to receive the corresponding acknowledgments.

/messages/devicebound

2. Custom Endpoints

Custom endpoints mainly deal with service endpoints and D2C messages. Instead of using built-in service endpoint (/messages/events) you can create custom endpoints. Custom endpoints can be on even hub, service bus queue or service bus topics

Azure IoT hub provides following type of custom endpoints each cater different requirement

  • Event Hubs
    • Event Hubs is a scalable event processing service that ingests and processes large volumes of events, with low latency and high reliability. Events in even hub can be retained for one to seven days. Events can be played back again. You can read more about event hub here
  • Service Bus Queues. Service bus messaging contains:
    • Queues, which allow one-directional communication. Each queue acts as an intermediary (sometimes called a broker) that stores sent messages until they are received. Each message is received by a single recipient
    • Service Bus Topics. Topics, which provide one-directional communication using subscriptions-a single topic can have multiple subscriptions. Like a queue, a topic acts as a broker, but each subscription can optionally use a filter to receive only messages that match specific criteria

 

Routes

Routes routes D2C messages to service endpoints in IoT hub. If there are no routes configured messages are routed to default service endpoint (/messages/events). Messages can be routed to custom endpoints also, check this link for routing rules

azure_iot_39

 

As seen in the above picture messages received by IoT hub is checked against route, if there are no route configured or none matches the rule, message routes to default endpoint (messages/events) otherwise message routed to respective endpoints

Pre-requisite

  1. Microsoft Visual Studio 2017. Link to install visual Studio here
  2. Account in Azure IoT hub
  3. Ubuntu 16.04 to simulate IoT device

Procedure

In this lab we will try these exercises

  1. Read D2C messages using built-in endpoint (/messages/events)
  2. Send C2D messages using built-in endpoint (/messages/devicebound)
  3. Read D2C messages using service bus queue
  4. Send C2D messages using service bus queue

Lot’s to do so let’s get started

1. Reading D2C messages using built-in endpoint (messages/events)

D2C messages are routed to default service endpoint (messages/events) if there is no route created in IoT hub. messages/events is Event hub compatible endpoint. We are not creating any route in this exercise so we can use this service endpoint to read D2C messages. This is the built-in endpoint in IoT hub so no need to create one.

You can find this endpoint under IoT Hub -> Endpoints

azure_iot_18

In Visual Studio, add a Visual C# Windows Classic Desktop project to the current solution, by using the Console App (.NET Framework) project template. Name the project readD2CMessage

azure_iot_26

In Solution Explorer, right-click the readD2CMessage project, and then click Manage NuGet Packages. This operation displays the NuGet Package Manager window.

Browse for WindowsAzure.ServiceBus, click Install, and accept the terms of use. This operation downloads, installs, and adds a reference to the Azure Service Bus, with all its dependencies.

azure_iot_27

Add the following using statements at the top of the Program.cs file:

using Microsoft.ServiceBus.Messaging;

Add your IoT connection string and default built-in service endpoint (/messages/events)

static string connectionString = "<your IoT hub connection string>";        
static string iotHubD2cEndpoint = "messages/events";

complete code looks like this

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.ServiceBus.Messaging;
using System.Threading;
using System.IO;

namespace ConsoleApp3
{
   class Program
   {
       static void Main(string[] args)
       {
           Console.WriteLine("Receive messages. Ctrl-C to exit.\n");
           eventHubClient = EventHubClient.CreateFromConnectionString(connectionString, iotHubD2cEndpoint);
         var d2cPartitions = eventHubClient.GetRuntimeInformation().PartitionIds;
           CancellationTokenSource cts = new CancellationTokenSource();
           System.Console.CancelKeyPress += (s, e) =>
           {
               e.Cancel = true;
               cts.Cancel();
               Console.WriteLine("Exiting...");
           };
           var tasks = new List<Task>();
           foreach (string partition in d2cPartitions)
           {
               tasks.Add(ReceiveMessagesFromDeviceAsync(partition, cts.Token));
           }
           Task.WaitAll(tasks.ToArray());
       }
       static string connectionString = "HostName=myIoT-Hub.azure-devices.net;SharedAccessKeyName=iothubowner;SharedAccessKey=xxxx=";
       static string iotHubD2cEndpoint = "messages/events";
       static EventHubClient eventHubClient;
       private static async Task ReceiveMessagesFromDeviceAsync(string partition, CancellationToken ct)
       {
           var eventHubReceiver = eventHubClient.GetDefaultConsumerGroup().CreateReceiver(partition, DateTime.UtcNow);
           while (true)
           {
               if (ct.IsCancellationRequested) break;
               EventData eventData = await eventHubReceiver.ReceiveAsync();
               if (eventData == null) continue;
               string data = Encoding.UTF8.GetString(eventData.GetBytes());
               var prop = (string)eventData.Properties["temperatureAlert"];
		Console.WriteLine("Message received. Partition: {0} Data: '{1}' Property: '{2}'", partition, data, prop);
           }
       }
   }
}

Now you are ready to run the applications.

Press F5 to start the console app. The App display D2C messages.

Start IoT device simulation program in your VM as we did in Lab-50.

You will see D2C messages on Console

azure_iot_21

2. Sending C2D messages using built-in endpoint (/messages/devicebound)

In Visual Studio, add a Visual C# Windows Classic Desktop project to the current solution, by using the Console App (.NET Framework) project template. Name the project sendC2DMessage

azure_iot_22

In Solution Explorer, right-click the sendC2DMessage project, and then click Manage NuGet Packages. This operation displays the NuGet Package Manager window.

Browse for Microsoft.Azure.Devices, click Install, and accept the terms of use. This operation downloads, installs, and adds a reference to the Azure Devices, with all its dependencies.

azure_iot_23

azure_iot_24

Add the following using statements at the top of the Program.cs file:

using Microsoft.Azure.Devices;

Add IoT hub connection string

static string connectionString = "<your IoT hub connection string>";

Add your device ID, in my case it is myIotDevice

await serviceClient.SendAsync("myIotDevice", commandMessage);

complete code looks like this

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Azure.Devices;

namespace sendC2DMessage
{
   class Program
   {
       static ServiceClient serviceClient;
       static string connectionString = "HostName=myIoT-Hub.azure-devices.net;SharedAccessKeyName=iothubowner;SharedAccessKey=xxxxxxxxxxxxx=";
       static void Main(string[] args)
       {
           Console.WriteLine("Send Cloud-to-Device message\n");
           serviceClient = ServiceClient.CreateFromConnectionString(connectionString);
           Console.WriteLine("Press any key to send a C2D message.");
           Console.ReadLine();
           sendC2DMessage().Wait();
           Console.ReadLine();
       }
       private async static Task sendC2DMessage()
       {
           var commandMessage = new Message(Encoding.ASCII.GetBytes("This is Cloud to Device message.."));
            await serviceClient.SendAsync("myIotDevice", commandMessage);
       }
   }
}

Now you are ready to run the applications.

Press F5 to start the console app.

Start IoT device simulation program in your VM as we did in Lab-50

Every time you hit enter on console app it will send C2D message “This is Cloud to Device message..” which will be visible in IoT device

3. Read D2C message using service bus queue endpoint

In this exercise we will route D2C messages to custom endpoint which is service bus queue and the read messages from endpoint

  1. Create service bus resource, New -> Enterprise Integration -> Service Bus

azure_iot_28

2.  Give service bus resource a name and attach to existing resource group

azure_iot_29

3. Note down service bus queue connection string, All resources -> <service bus> -> Shared access policies -> RootManagerSharedAccessKey -> Primary Connection String

azure_iot_30

4. Create a queue inside service bus resource, <service bus> -> Queues. Give queue name and click Create

azure_iot_31

5. Create an endpoint in IoT hub, click on All Resource -> IoT hub -> Endpoint -> Add. Select endpoint type as ‘Service Bus Queue’,  select Service Bus name in Service Bus namespace and finally select queue name

azure_iot_32

In Visual Studio, add a Visual C# Windows Classic Desktop project to the current solution, by using the Console App (.NET Framework) project template. Name the project receiveD2CMessageFromQueue

In Solution Explorer, right-click the receiveD2CMessageFromQueue project, and then click Manage NuGet Packages. This operation displays the NuGet Package Manager window.

Browse for WindowsAzure.Servicebus, click Install, and accept the terms of use. This operation downloads, installs, and adds a reference to the Azure Devices, with all its dependencies.

Add the following using statements at the top of the Program.cs file:

using System.IO;
using Microsoft.ServiceBus.Messaging;

Add your service bus queue connection string saved in earlier step. Provide your queue name

const string ServiceBusConnectionString = "<your service bus queue connection string>";      
const string QueueName = "<your queue name>";

Complete code looks like this

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.IO;
using Microsoft.ServiceBus.Messaging;

namespace readD2CMessageFromQueue
{
  class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Receive critical messages. Ctrl-C to exit.\n");
            var connectionString = "Endpoint=sb://divinesvcbus.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=xxxxxx=";
            var queueName = "divineQueue";
            var client = QueueClient.CreateFromConnectionString(connectionString, queueName);
            client.OnMessage(message =>
            {
                Stream stream = message.GetBody<Stream>();
                StreamReader reader = new StreamReader(stream, Encoding.ASCII);
                string s = reader.ReadToEnd();
                Console.WriteLine(String.Format("Message body: {0}", s));
            });

           Console.ReadLine();
        }
    }
}

Now you are ready to run the applications.

Press F5 to start the console app.

Start IoT device simulation program in your VM as we did in Lab-50

You will see D2C messages on console

azure_iot_33

Let’s tweak the routing rule to route messages to queue endpoint only when temperatureAlert = true. The logic in simulated device (Ubuntu VM) is to send temperatureAlert = true whenever temp > 28. More on creating routing query here

Edit route  to filter based on temperatureAlert = true. All Resources -> IoT hub -> Route and add following in query temperatureAlert = ‘true’

azure_iot_34

Start console app and simulated device. Now you should see only messages with temperature > 28. This mean our routing rule is working

azure_iot_35

 

Lab-39: Ansible for Network Automation

The goal of this lab is to learn Ansible basics and explore how it can help automate network device.

Ansible is a configuration management tool for servers. It is in the same line of tools as Chef, Puppet and Salt. The difference is, Ansible is agentless which mean it does not require any agent running on the server you are trying to configure. Which is a huge plus because network devices like switches and routers can’t be loaded with any agent

Ansible does it’s job by executing modules. Modules are python libraries. Ansible has a wide range of module . Click here  to lean about Ansible modules.

Ansible is basically made of these two components

  1. Modules: programs to perform a task
  2. Inventory file: This file contains remote server info like IP address, ssh connection info

When you install Ansible modules are loaded as part of installation. In my environment modules are located in this directory

/usr/lib/python2.7/site-packages/ansible/modules

To check what modules installed in your machine try this

$ansible-doc -l

To check the detail of a module try this

$ansible-doc -s $ansible-doc -s file

Prerequisite:

Install Ansible

$sudo pip install ansible

I am using Centos 7.3 for this lab. This is my Ansible version

$ansible --version
ansible 2.2.1.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides

Topology:

I have two bare metal servers. Ansible is installed on one server (Ansible server) and another server (remote server) used for configuration

Procedure:

Setup passwordless ssh

Let’s setup  passwordless access to remote server so we don’t have to type password every time Ansible executed. Ansible prefer to login using ssh keys. To setup passwordless access try these commands on Ansible server and remote server.

In this example I am using username:virtuora and password:virtuora on remote server

1. Generate ssh-key on Ansible server as well as on remote server for a user.
$ssh-keygen -t rsa

2. Now copy remote server public ssh-key to Ansible server and Ansible server 
public key to remote server
$ssh-copy-id -i ~/.ssh/id_rsa.pub virtuora@192.254.211.168

3. Test it out by ssh to remote server from Ansible server and make sure 
ssh works passwordless
$ssh virtuora@192.254.211.168

Inventory File

Inventory file contains remote server reachability info (IP address, ssh user, port number etc). It is a simple text file with remote server IP address and optionally can contain ssh info. I named my inventory file ‘hosts’ . I have only one remote server (vnc-server) to configure

[vnc-server]
192.254.211.168 ansible_connection=ssh ansible_port=22 ansible_user=virtuora

If you have multiple remote servers you can group them like this

[db-servers]
192.254.211.166
192.254.211.167

[web-server]
192.254.211.165

There are two ways to execute Ansible

  1. Adhoc, which is Ansible command line
  2. Ansible playbook, which is essentially yaml with Jinja2 template

Ansible with adhoc

This is the simplest Ansible adhoc command. It is using localhost so inventory file is not needed

$ansible all -i "localhost," -c local -m shell -a 'echo hello world'
localhost | SUCCESS | rc=0 >>
hello world

Try below adhoc command with ping module. This is not a traditional ICMP ping. If successful it mean Ansible server can login to remote server and remote server has usable python configured

$ansible -i hosts -m ping vnc-server

-i: specify the inventory file, in this case  ‘hosts’

-m: Ansible module name, in this case ping

vnc-server: This is the remote server name in inventory file

$ansible -i hosts -m ping vnc-server
192.254.211.168 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

Now try same command with increase  verbosity -vvv flag, it will help us understand Ansible internal. If you study log you will find that Ansible first sftp the ping module from Ansible server to remote server, executes the module on remote server and then deletes it

Note: In a sense Ansible is not completely agent less. It requires remote server to support sftp and python to execute module, this could be a problem for network devices (switches & routers)  which doesn’t have these capabilities

$ansible -i hosts -m ping vnc-server -vvv
Using /etc/ansible/ansible.cfg as config file
Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/system/ping.py
 ESTABLISH SSH CONNECTION FOR USER: virtuora
 SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=virtuora -o ConnectTimeout=10 -o ControlPath=/home/divine/.ansible/cp/ansible-ssh-%h-%p-%r 192.254.211.168 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1491254258.92-242743076706801 `" && echo ansible-tmp-1491254258.92-242743076706801="` echo ~/.ansible/tmp/ansible-tmp-1491254258.92-242743076706801 `" ) && sleep 0'"'"''
 PUT /tmp/tmpXP3MNY TO /home/virtuora/.ansible/tmp/ansible-tmp-1491254258.92-242743076706801/ping.py
 SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=virtuora -o ConnectTimeout=10 -o ControlPath=/home/divine/.ansible/cp/ansible-ssh-%h-%p-%r '[192.254.211.168]'
 ESTABLISH SSH CONNECTION FOR USER: virtuora
 SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=virtuora -o ConnectTimeout=10 -o ControlPath=/home/divine/.ansible/cp/ansible-ssh-%h-%p-%r 192.254.211.168 '/bin/sh -c '"'"'chmod u+x /home/virtuora/.ansible/tmp/ansible-tmp-1491254258.92-242743076706801/ /home/virtuora/.ansible/tmp/ansible-tmp-1491254258.92-242743076706801/ping.py && sleep 0'"'"''
 ESTABLISH SSH CONNECTION FOR USER: virtuora
 SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=virtuora -o ConnectTimeout=10 -o ControlPath=/home/divine/.ansible/cp/ansible-ssh-%h-%p-%r -tt 192.254.211.168 '/bin/sh -c '"'"'/usr/bin/python /home/virtuora/.ansible/tmp/ansible-tmp-1491254258.92-242743076706801/ping.py; rm -rf "/home/virtuora/.ansible/tmp/ansible-tmp-1491254258.92-242743076706801/" > /dev/null 2>&1 && sleep 0'"'"''
192.254.211.168 | SUCCESS => {
    "changed": false,
    "invocation": {
        "module_args": {
            "data": null
        },
        "module_name": "ping"
    },
    "ping": "pong"
}

There are some variations to adhoc command, say if you want to specify user on command line instead of adding in inventory file (as I did)  you can try this

In this case I specify user using -u option

$ansible -i hosts -m ping vnc-server -u virtuora
167.254.211.168 | SUCCESS => {
    "changed": false,
    "ping": "pong"

Privilege Escalation

Privilege escalation allows to become another user that we login with. It can be  useful when we need to become sudo for some commands.

Say if you want to install a package on remote server which require sudo access. You can run it this way:

$ansible -i hosts -m yum -a “name=bridge-utils state=present” vnc-server –become -K

–become: privilege escalation is true

-K: ask password for SUDO

$ansible -i hosts -m yum -a "name=bridge-utils state=present" vnc-server --become -K
SUDO password:
167.254.211.168 | SUCCESS => {
    "changed": false,
    "msg": "",
    "rc": 0,
    "results": [
        "bridge-utils-1.5-9.el7.x86_64 providing bridge-utils is already installed"
    ]
}

Ansible Playbook

Running Ansible on command line is not very efficient, playbook allows you to run multiple tasks at once. You can share playbook with other users . A playbook contains multiple tasks. Playbook is written in yaml format with jinja2 template. To learn basics of playbook click here

This is a simple playbook which installs git package on remote server using yum. In this playbook I am defining variable (vars) and privilege escalation using become:true and become_method:sudo

$ cat yum-playbook.yml
---
- hosts: vnc-server
  vars:
    package_name: git
  tasks:
   - name: Install git package
     yum:
      state: present
      name: "{{ package_name }}"
     become: true
     become_method: sudo

hosts: remote server name on inventory file
vars: variable definition
become: privilege escalation
become_method: privilege escalation method

Execute playbook with -K to prompt for sudo password

$ansible-playbook -i hosts yum-playbook.yml -K
SUDO password:

PLAY [vnc-server] **************************************************************

TASK [setup] *******************************************************************
ok: [192.254.211.168]

TASK [Install git package] *****************************************************
changed: [192.254.211.168]

PLAY RECAP *********************************************************************
192.254.211.168            : ok=2    changed=1    unreachable=0    failed=0

changed=1 mean one change applied to remote server, in this case git package installed

Run it again

$ansible-playbook -i hosts yum-playbook.yml -K
SUDO password:

PLAY [vnc-server] **************************************************************

TASK [setup] *******************************************************************
ok: [192.254.211.168]

TASK [Install git package] *****************************************************
ok: [192.254.211.168]

PLAY RECAP *********************************************************************
192.254.211.168            : ok=2    changed=0    unreachable=0    failed=0

This time change=0 because git package was already present so no action performed on remote server. This is Ansible idempotent behavior, which mean  it performs action only when needed.

Configure Network device using Ansible

Configuring servers and all is good but I am interested in configuring  network devices like switches, routers. I like to know what Ansible can do for these devices. In my case I have an optical network switch which doesn’t support python or sftp. It does support ssh.

As I mentioned earlier Ansible modules need sftp and python configured on remote server. Let’s see how to configure an optical switch using  these constraint.

Ansible has a module called ‘raw’, read more about it here. This is the only module I found which doesn’t require sftp and python on remote server. This module sends commands on open ssh connection it doesn’t copy module to remote server.

This is my playbook with raw module looks like. This playbook provisions ports in my optical switch.

This playbook has two tasks 1) configure ports 2) set fail when port configuration task fail

In this playbook I have defined a dictionary of shelf/slot/port, configure ports task iterate through this dictionary to configure ports. I am also using ‘register’ to register output of configure ports task, this will  be used in next task to set ‘failed’ field

gather_facts: false is important here so Ansible doesn’t try to gather facts from our network device as it does on server. This parameter is true by default.

---
- hosts: optical-switch
  remote_user: virtuora
  gather_facts: false
  vars:
      ports:
           P1:
              slot_no: 2
              port_no: 7
              port_type: 10GER
           P2:
              slot_no: 2
              port_no: 8
              port_type: 10GER
           P3:
              slot_no: 2
              port_no: 9
              port_type: 10GER

      shelf_no: 1
      subslot_no: 0
  tasks:
   - name: configure ports
     raw: |
       configure
       set eqpt shelf "{{ shelf_no }}" slot {{ item.value.slot_no }} subslot "{{ subslot_no }}" port {{ item.value.port_no }}  pluggableInterfaceType {{ item.value.port_type }} admin-status up
       commit
     register: port_config
     with_dict: "{{ ports }}"

   - name: Set fail when port configuration fail
     fail:
        msg: "Configure ports failed"
     when: item.stdout.find('error') != -1
     with_items: "{{ port_config.results }}"
     #debug: msg="{{ port_config }}"

Execute the playbook. I don’t have password less ssh to device so -k used to provide password during run time

$ ansible-playbook -i hosts playbook-s100.yml -k
SSH password:

PLAY [s100-1] ******************************************************************

TASK [configure ports] *********************************************************
changed: [192.254.210.33] => (item={'key': u'P2', 'value': {u'slot_no': 2, u'port_no': 8, u'port_type': u'10GER'}})
changed: [192.254.210.33] => (item={'key': u'P3', 'value': {u'slot_no': 2, u'port_no': 9, u'port_type': u'10GER'}})
changed: [192.254.210.33] => (item={'key': u'P1', 'value': {u'slot_no': 2, u'port_no': 7, u'port_type': u'10GER'}})

TASK [Set pass/fail] ***********************************************************
skipping: [192.254.210.33] => (item={u'changed': True, u'_ansible_no_log': False, u'stdout': u'Commit complete.\r\n', u'_ansible_item_result': True, u'item': {u'key': u'P2', u'value': {u'slot_no': 2, u'port_no': 8, u'port_type': u'10GER'}}, u'stderr': u'\nWelcome to the FUJITSU 1FINITY S100\nCopyright Fujitsu Limited.\n\nShared connection to 192.254.210.33 closed.\r\n', u'rc': 0, u'invocation': {u'module_name': u'raw', u'module_args': {u'_raw_params': u'configure\n set eqpt shelf "1" slot 2 subslot "0" port 8 pluggableInterfaceType 10GER admin-status up\n commit'}}, u'stdout_lines': [u'Commit complete.']})
skipping: [192.254.210.33] => (item={u'changed': True, u'_ansible_no_log': False, u'stdout': u'Commit complete.\r\n', u'_ansible_item_result': True, u'item': {u'key': u'P3', u'value': {u'slot_no': 2, u'port_no': 9, u'port_type': u'10GER'}}, u'stderr': u'Shared connection to 192.254.210.33 closed.\r\n', u'rc': 0, u'invocation': {u'module_name': u'raw', u'module_args': {u'_raw_params': u'configure\n set eqpt shelf "1" slot 2 subslot "0" port 9 pluggableInterfaceType 10GER admin-status up\n commit'}}, u'stdout_lines': [u'Commit complete.']})
skipping: [192.254.210.33] => (item={u'changed': True, u'_ansible_no_log': False, u'stdout': u'Commit complete.\r\n', u'_ansible_item_result': True, u'item': {u'key': u'P1', u'value': {u'slot_no': 2, u'port_no': 7, u'port_type': u'10GER'}}, u'stderr': u'Shared connection to 192.254.210.33 closed.\r\n', u'rc': 0, u'invocation': {u'module_name': u'raw', u'module_args': {u'_raw_params': u'configure\n set eqpt shelf "1" slot 2 subslot "0" port 7 pluggableInterfaceType 10GER admin-status up\n commit'}}, u'stdout_lines': [u'Commit complete.']})

PLAY RECAP *********************************************************************
192.254.210.33             : ok=1    changed=1    unreachable=0    failed=0

Playbook ran fine and configured three ports on optical switch.

Execute playbook again

$ ansible-playbook -i hosts playbook-s100.yml -k
SSH password:

PLAY [s100-1] ******************************************************************

TASK [configure ports] *********************************************************
changed: [192.254.210.33] => (item={'key': u'P2', 'value': {u'slot_no': 2, u'port_no': 8, u'port_type': u'10GER'}})
changed: [192.254.210.33] => (item={'key': u'P3', 'value': {u'slot_no': 2, u'port_no': 9, u'port_type': u'10GER'}})
changed: [192.254.210.33] => (item={'key': u'P1', 'value': {u'slot_no': 2, u'port_no': 7, u'port_type': u'10GER'}})

TASK [Set pass/fail] ***********************************************************
failed: [192.254.210.33] (item={u'changed': True, u'_ansible_no_log': False, u'stdout': u'Error: access denied\r\n[error][2017-04-10 17:04:47]\r\n', u'_ansible_item_result': True, u'item': {u'key': u'P2', u'value': {u'slot_no': 2, u'port_no': 8, u'port_type': u'10GER'}}, u'stderr': u'Shared connection to 192.254.210.33 closed.\r\n', u'rc': 0, u'invocation': {u'module_name': u'raw', u'module_args': {u'_raw_params': u'configure\n set eqpt shelf "1" slot 2 subslot "0" port 8 pluggableInterfaceType 10GER admin-status up\n commit'}}, u'stdout_lines': [u'Error: access denied', u'[error][2017-04-10 17:04:47]']}) => {"failed": true, "item": {"changed": true, "invocation": {"module_args": {"_raw_params": "configure\n set eqpt shelf \"1\" slot 2 subslot \"0\" port 8 pluggableInterfaceType 10GER admin-status up\n commit"}, "module_name": "raw"}, "item": {"key": "P2", "value": {"port_no": 8, "port_type": "10GER", "slot_no": 2}}, "rc": 0, "stderr": "Shared connection to 192.254.210.33 closed.\r\n", "stdout": "Error: access denied\r\n[error][2017-04-10 17:04:47]\r\n", "stdout_lines": ["Error: access denied", "[error][2017-04-10 17:04:47]"]}, "msg": "The command failed"}
failed: [192.254.210.33] (item={u'changed': True, u'_ansible_no_log': False, u'stdout': u'Error: access denied\r\n[error][2017-04-10 17:04:47]\r\n', u'_ansible_item_result': True, u'item': {u'key': u'P3', u'value': {u'slot_no': 2, u'port_no': 9, u'port_type': u'10GER'}}, u'stderr': u'Shared connection to 192.254.210.33 closed.\r\n', u'rc': 0, u'invocation': {u'module_name': u'raw', u'module_args': {u'_raw_params': u'configure\n set eqpt shelf "1" slot 2 subslot "0" port 9 pluggableInterfaceType 10GER admin-status up\n commit'}}, u'stdout_lines': [u'Error: access denied', u'[error][2017-04-10 17:04:47]']}) => {"failed": true, "item": {"changed": true, "invocation": {"module_args": {"_raw_params": "configure\n set eqpt shelf \"1\" slot 2 subslot \"0\" port 9 pluggableInterfaceType 10GER admin-status up\n commit"}, "module_name": "raw"}, "item": {"key": "P3", "value": {"port_no": 9, "port_type": "10GER", "slot_no": 2}}, "rc": 0, "stderr": "Shared connection to 192.254.210.33 closed.\r\n", "stdout": "Error: access denied\r\n[error][2017-04-10 17:04:47]\r\n", "stdout_lines": ["Error: access denied", "[error][2017-04-10 17:04:47]"]}, "msg": "The command failed"}
failed: [192.254.210.33] (item={u'changed': True, u'_ansible_no_log': False, u'stdout': u'Error: access denied\r\n[error][2017-04-10 17:04:48]\r\n', u'_ansible_item_result': True, u'item': {u'key': u'P1', u'value': {u'slot_no': 2, u'port_no': 7, u'port_type': u'10GER'}}, u'stderr': u'Shared connection to 192.254.210.33 closed.\r\n', u'rc': 0, u'invocation': {u'module_name': u'raw', u'module_args': {u'_raw_params': u'configure\n set eqpt shelf "1" slot 2 subslot "0" port 7 pluggableInterfaceType 10GER admin-status up\n commit'}}, u'stdout_lines': [u'Error: access denied', u'[error][2017-04-10 17:04:48]']}) => {"failed": true, "item": {"changed": true, "invocation": {"module_args": {"_raw_params": "configure\n set eqpt shelf \"1\" slot 2 subslot \"0\" port 7 pluggableInterfaceType 10GER admin-status up\n commit"}, "module_name": "raw"}, "item": {"key": "P1", "value": {"port_no": 7, "port_type": "10GER", "slot_no": 2}}, "rc": 0, "stderr": "Shared connection to 192.254.210.33 closed.\r\n", "stdout": "Error: access denied\r\n[error][2017-04-10 17:04:48]\r\n", "stdout_lines": ["Error: access denied", "[error][2017-04-10 17:04:48]"]}, "msg": "The command failed"}
        to retry, use: --limit @/home/divine/ansible/vnc_install/playbook-s100.retry

PLAY RECAP *********************************************************************
192.254.210.33             : ok=1    changed=1    unreachable=0    failed=1

As you can see failed=1 because port configuration denied due to existing provisionig

I didn’t find other Ansible modules which can help to configure my device. However if you are using Juniper or Cisco you are at luck they have written special Ansible modules for their devices

Ansible has support for many situations. Here is what I learned

What if  you need to prompt user before running playbook tasks and exit if confirmation fail

- hosts: vnc-server
  vars_prompt:
     name: "confir"
     prompt: "Are you sure you want to un-install VNC, answer with 'yes'"
     default: "no"
     private: no
  tasks:

   - name: Check Confirmation
     fail: msg="confirmation failed"
     when: confirm != "yes"

How about if you need to execute a task based on output of previous task. In this example I am checking the status of ‘vnc’ application and if it is not running then only start it in second task

- name: Check VNC status
  command: vnc status
  register: vnc_status

- name: start vnc if not already running
  shell: nohup vnc start
  when: item.find('DOWN')
  with_items: "{{ vnc_status.stdout }}"

If your application doesn’t have way to check status, you can check status using Linux ‘ps’ command. Write a simple shell command like this to check application status

  - name: check vnc status
    shell: if ps -ef | egrep 'karaf' | grep -v grep > /dev/null; then echo "vnc_running"; else echo "vnc_not_running"; fi
    register: vnc_status

  - name: start vnc if not already running
    shell: nohup vnc start
    when: vnc_status.stdout.find('vnc_not_running') == 0

Ansible gather facts from remote server. You can use these facts  in your playbook. In this example I am using ‘ansible_pkg_mgr’ which will be either yum or apt-get depending on your Linux version

 - name: Install git package
     yum:
      state: present
      name: git
     when: ansible_pkg_mgr == "yum"
     become: true
     become_method: sudo

You can check facts gathered by Ansibile by running this command

ansible -i hosts -m setup vnc-server

 

Lab-45: Disaggregation

Disaggregation has become the buzz word recently. Working in telecom field  for 20+ years I have seen lots of changes  in telecom products.  This is my take on this subject. I mainly worked in optical transport products and transport technologies like SONET/SDH, Carrier Ethernet etc.

When I  joined telecom world I worked on products which were built for single technology, for example we had one box for SONET another for ATM ,  DWDM etc. The interfaces were proprietary and tightly coupled network management system (NMS). For the capacity size of the boxes were large and port density was low.

This is the picture of one such box

fujitsu_4100

Then around 2003-2004 we built a box which combined all transport technologies in one, industry called it the GOD box. This box combined technologies like SONET, OTN, ROADM and Carrier Ethernet into one. The hardware capacity improved, density improved we could pack more ports in one card. The size of chassis became larger. The management interfaces were still proprietary and tightly coupled network management system.

This was a real complicated box. Development and testing was a challenge. SW was tightly coupled, a single change in one technology require regression testing of all technologies. A centralized switch fabric serving different technologies was a challenge.

Here is the picture of one such box

fujitsu_9500

In last 2 years around 2013-14 disaggregation became popular and the whole industry starting to move in this direction. The GOD box broken down into small boxes. It’s like Ma Bell split into baby Bell companies. GOD box split into smaller pizza boxes SONET/SDH, Carrier Ethernet (Packet over Ethernet), OTN, ROADM etc. Hardware became commodity .

On the software side lots of changes happened. Development environment switched from embedded OS to Linux based. Developers started to use lots of open source. Interface starting to become open with Netconf and Yang data modelling.  OPENROADM project kicked in by ATT opening door for multi-vendor interoperability

On the network management side SDN starting to influence separation of control plane from data plane. Architects started talking about how to move control functionality out of the box.

Product development methodology changed from typical waterfall to Agile based, from a monolithic to component based sw development. Developing common component and reusing them on multiple products. Development tools changed. Management expectation changed and started demanding shorter product life cycle. More frequent releases. Customer engagement increased more demos and MVP loads.

Here are the  picture of disaggregated boxes

fujitsu_1finity

So what is fueling disaggregation. In my view it is a combination of hardware and software . Hardware is becoming commodity and in the software side new technologies like SDN and NFV

Lab-41: Create Multi-node infrastructure using Vagrant

I will create vagrant multi-node configuration and use it for my future labs. These nodes have very basic provisioning like hostname, ip and some useful packages.

It is really simple to setup VMs using vagrant. You can read more about vagrant here.In this lab I am creating 3 VM configuration using vagrant box centos/7. I will add basic packages and then save them as vagrant boxes for future use.

you check vagrant registry for boxes here.

Below is the procedure I followed to build my multi-node infrastructure

  1. Download and install Vagrant. I am using Centos 7 host machine select vagrant package according to your machine here
  2. Install vagrant package
$ sudo rpm -Uvh vagrant_1.9.1_x86_64.rpm
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

[sudo] password for divine:
Preparing...                          ################################# [100%]
Updating / installing...
   1:vagrant-1:1.9.1-1                ################################# [100%]

3. Install Virtual box. Below is the procedure from official Centos site. I am using virtualbox for my VMs, it is a default provider for vagrant other options are vmware,hyper-v and docker

$sudo yum update -y
$sudo yum install dkms
$sudo wget -P /etc/yum.repos.d http://download.virtualbox.org/virtualbox/rpm/rhel/virtualbox.repo
$sudo yum install VirtualBox-5.0
$sudo usermod -a -G vboxusers <your_user_name>

4. Create vagrantfile in the directory where you want to work from. Note: vagrant checks current directory for vagrantfile

This command will create a generic vagrantfile.

$vagrant init

5. Open  vagrantfile and add below commands. These commands will spin 3 VMs. I have named them Master, Node1 & Node2. Each VM assigned 2048 MB RAM, 2 CPU core, IP address and hostname.

As can be seen I am using centos/7 in vm.box

$ cat Vagrantfile
Vagrant.configure("2") do |config|

#VM1: Master
 config.vm.define "master" do |master|
 master.vm.box = "centos/7"
 master.vm.hostname = "Master"
 master.vm.network :private_network, ip: "192.168.11.11"
 master.ssh.insert_key = false
 config.vm.provider :virtualbox do |v|
    v.customize ["modifyvm", :id, "--memory", 2048]
    v.customize ["modifyvm", :id, "--cpus", 2]
 end
end

#VM2: Node1
 config.vm.define "node1" do |node1|
 node1.vm.box = "centos/7"
 node1.vm.hostname = "Node1"
 node1.vm.network :private_network, ip: "192.168.11.12"
 node1.ssh.insert_key = false
 config.vm.provider :virtualbox do |v|
   v.customize ["modifyvm", :id, "--memory", 2048]
   v.customize ["modifyvm", :id, "--cpus", 2]
 end
end

#VM3: Node2
 config.vm.define "node2" do |node2|
 node2.vm.box = "centos/7"
 node2.vm.hostname = "Node2"
 node2.vm.network :private_network, ip: "192.168.11.13"
 node2.ssh.insert_key = false
 config.vm.provider :virtualbox do |v|
   v.customize ["modifyvm", :id, "--memory", 2048]
   v.customize ["modifyvm", :id, "--cpus", 2]
 end
end
end

5. Bring up VMs. This command will spin VMs, setup two network interface 1) eth0:NAT and 2)eth1:host only with IP address provided in vagrantfile

$ vagrant up
Bringing machine 'master' up with 'virtualbox' provider...
Bringing machine 'node1' up with 'virtualbox' provider...
Bringing machine 'node2' up with 'virtualbox' provider...
==> master: Importing base box 'centos/7'...
==> master: Matching MAC address for NAT networking...
==> master: Checking if box 'centos/7' is up to date...
==> master: Setting the name of the VM: kubernetes_master_1487800004247_23604
==> master: Clearing any previously set network interfaces...
==> master: Preparing network interfaces based on configuration...
    master: Adapter 1: nat
    master: Adapter 2: hostonly
==> master: Forwarding ports...
    master: 22 (guest) => 2222 (host) (adapter 1)
==> master: Booting VM...
==> master: Waiting for machine to boot. This may take a few minutes...
    master: SSH address: 127.0.0.1:2222
    master: SSH username: vagrant
    master: SSH auth method: private key
    master:
    master: Vagrant insecure key detected. Vagrant will automatically replace
    master: this with a newly generated keypair for better security.
    master:
    master: Inserting generated public key within guest...
    master: Removing insecure key from the guest if it's present...
    master: Key inserted! Disconnecting and reconnecting using new SSH key...
==> master: Machine booted and ready!
==> master: Checking for guest additions in VM...
    master: No guest additions were detected on the base box for this VM! Guest
    master: additions are required for forwarded ports, shared folders, host only
    master: networking, and more. If SSH fails on this machine, please install
    master: the guest additions and repackage the box to continue.
    master:
    master: This is not an error message; everything may continue to work properly,
    master: in which case you may ignore this message.
==> master: Setting hostname...
==> master: Configuring and enabling network interfaces...

<--------> output truncated

6. Check VM status

//this command gives ssh port info for each VM
$ vagrant ssh-config
Host master
  HostName 127.0.0.1
  User vagrant
  Port 2222
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /home/divine/vagrant/proj/kubernetes/.vagrant/machines/master/virtualbox/private_key
  IdentitiesOnly yes
  LogLevel FATAL

Host node1
  HostName 127.0.0.1
  User vagrant
  Port 2200
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /home/divine/vagrant/proj/kubernetes/.vagrant/machines/node1/virtualbox/private_key
  IdentitiesOnly yes
  LogLevel FATAL

Host node2
  HostName 127.0.0.1
  User vagrant
  Port 2201
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /home/divine/vagrant/proj/kubernetes/.vagrant/machines/node2/virtualbox/private_key
  IdentitiesOnly yes
  LogLevel FATAL

$ vagrant status
Current machine states:
master                    running (virtualbox)
node1                     running (virtualbox)
node2                     running (virtualbox)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.

7. login to VMs do housekeeping and  load packages. Below example of my master node

Note:The root password for vagrant box is vagrant

//ssh to Master node
$vagrant ssh master

//bring up eth1 interface by default eth1 is down
[vagrant@Master ~]$ sudo ifup eth1

[vagrant@Master ~]$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:22:5b:53 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic eth0
       valid_lft 74600sec preferred_lft 74600sec
    inet6 fe80::5054:ff:fe22:5b53/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:f8:f1:21 brd ff:ff:ff:ff:ff:ff
    inet 192.168.11.11/24 brd 192.168.11.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fef8:f121/64 scope link
       valid_lft forever preferred_lft forever

//load some useful packages
{vagrant@Master ~]$ sudo yum update -y
[vagrant@Master ~]$ sudo yum install net-tools
[vagrant@Master ~]$ sudo yum install wget
[vagrant@Master ~]$ sudo yum

//setup ssh key
[vagrant@Master ~]$ wget https://raw.githubusercontent.com/mitchellh/vagrant/master/keys/vagrant.pub -O .ssh/authorized_keys
[vagrant@Master ~]$ chmod 700 .ssh
[vagrant@Master ~]$ chmod 600 .ssh/authorized_keys
[vagrant@Master ~]$ chown -R vagrant:vagrant .ssh

8. Package  VMs into vagrant box. I will create 3 boxes named: Master, Node1 & Node2

//we need virtualbox name to create box, below command will provide virtualbox
names
$ ps -ef | grep virtualbox
divine     366 19858  0 22:52 pts/2    00:00:00 grep --color=auto virtualbox
divine    9630     1  0 19:29 ?        00:00:14 /usr/lib/virtualbox/VBoxXPCOMIPCD
divine    9635     1  0 19:29 ?        00:00:39 /usr/lib/virtualbox/VBoxSVC --auto-shutdown
divine   10449  9635  1 19:29 ?        00:02:01 /usr/lib/virtualbox/VBoxHeadless --comment kubernetes_master_1487800004247_23604 --startvm c2acbdc8-457a-40e3-ac21-fe151b886ca7 --vrde config
divine   12633  9635  0 19:30 ?        00:01:52 /usr/lib/virtualbox/VBoxHeadless --comment kubernetes_node1_1487800067490_38506 --startvm 3bedc067-befc-4b81-9042-4a08c126ca45 --vrde config
divine   14735  9635  0 19:30 ?        00:01:51 /usr/lib/virtualbox/VBoxHeadless --comment kubernetes_node2_1487800127574_14083 --startvm cb10ab77-0940-4243-b01b-9ce2e3fcf2af --vrde config

//create box
$ vagrant package --base kubernetes_master_1487800004247_23604 --output Master --vagrantfile Vagrantfile
$ vagrant package --base kubernetes_node1_1487800067490_38506 --output Node1 --vagrantfile Vagrantfile
$ vagrant package --base kubernetes_node2_1487800127574_14083 --output Node2 --vagrantfile Vagrantfile

$ ls
Master  Node1  Node2  Vagrantfile

9.Now that I have created my boxes. I will destroy current VMs

$vagrant destroy

10. The last step is to modify  vagrantfile to use my newly minted boxes. The change is in vm.box parameter, instead of centos/7 I am providing my boxes name

$ cat Vagrantfile
Vagrant.configure("2") do |config|

#VM1: Master
 config.vm.define "master" do |master|
  master.vm.box = "Master"
  master.vm.hostname = "Master"
  master.vm.network :private_network, ip: "192.168.11.11"
  master.ssh.insert_key = false
  config.vm.provider :virtualbox do |v|
    v.customize ["modifyvm", :id, "--memory", 2048]
    v.customize ["modifyvm", :id, "--cpus", 2]
  end
 end

#VM2: Node1
 config.vm.define "node1" do |node1|
  node1.vm.box = "Node1"
  node1.vm.hostname = "Node1"
  node1.vm.network :private_network, ip: "192.168.11.12"
  node1.ssh.insert_key = false
  config.vm.provider :virtualbox do |v|
    v.customize ["modifyvm", :id, "--memory", 2048]
    v.customize ["modifyvm", :id, "--cpus", 2]
  end
 end

#VM3: Node2
 config.vm.define "node2" do |node2|
  node2.vm.box = "Node2"
  node2.vm.hostname = "Node2"
  node2.vm.network :private_network, ip: "192.168.11.13"
  node2.ssh.insert_key = false
  config.vm.provider :virtualbox do |v|
    v.customize ["modifyvm", :id, "--memory", 2048]
    v.customize ["modifyvm", :id, "--cpus", 2]
  end
 end
end

Lab-40: Container orchestration: Kubernetes

In Lab-35 I went over Docker Swarm, which is a container orchestration framework from Docker. In this lab I will go over another orchestration framework called Kubernetes. Kubernetes is an open source platform developed by Google. It provides orchestration for Docker and other types of containers

The purpose of this lab is to get familiar with Kubernetes, install it on Linux and deploy a simple Kubernetes pods. This lab is a  multi-node deployment of Kubernetes cluster.

Let’s get familiar with Kubernetes terminology

Master:

A Master is a VM or a physical computer responsible for managing the cluster. The master coordinates all activities in your cluster, such as scheduling applications, maintaining applications’ desired state, scaling applications, and rolling out new updates.

By default pods are not scheduled on Master. But if you like to schedule pods on Master try this command on Master

# kubectl taint nodes --all dedicated-

Node:

A node is a VM or a physical computer that serves as a worker machine in a Kubernetes cluster. Each node has a Kubelet, which is an agent for managing the node and communicating with the Kubernetes master. The node should also have tools for handling container operations, such as Docker.

Pod:

A pod is a  group of one or more containers. All the containers in a pod scheduled together, live together and die together. Why Kubernetes deploy pod and not containers because some applications are tightly coupled and make sense to deploy together i.e. web server and cache server. You can have separate containers for web server and cache server but deploy them together that way you make sure they are scheduled together on the same node and terminated together.It is easier  to manage pod than containers.  Read about pod here

Pod has similarity to VM in terms on process virtualization , both run multiple processes (in case of pod containers), all processes share same IP address, all processes can communicate using local host and they use separate network namespace then host

Some key points about pod:

  1. Containers is a pod are always co-located and co-scheduled, and run in a shared context
  2. Pod contains one or more application containers which are relatively tightly coupled — in a pre-container world, they would have executed on the same physical or virtual machine
  3. The shared context of a pod is a set of Linux namespaces, cgroups, and potentially other facets of isolation – the same things that isolate a Docker container
  4. Containers within a pod share an IP address, port space and hostname. Container within pod communicate using localhost
  5. Every pod get an IP address

Below example of pod deployment in a node.

kubernetes_5

Replication controller

Replication controller in Kubernetes is responsible for replicating pods. A ReplicationController ensures that a specified number of pod “replicas” are always running at any one time. It checks pod’s health and if a pod dies it quickly re-creates it automatically

API server

Kubernetes deploy API server on Master. API server provides front end to cluster. It serves REST services. You can interact with cluster using 1) cli (kubectl) 2)REST API 3)gui interface. kubectl & GUI internally uses REST API

kubernetes_6

Prerequisite:

In this lab I am using my 3 node vagrant infrastructure. Check  Lab-41 for detail how to setup VMs in vagrant. I have one Master and two Nodes. This is my VM topology

kubernetes_4

My VM specification

[root@Master ~]# cat /etc/*release*
CentOS Linux release 7.2.1511 (Core)
Derived from Red Hat Enterprise Linux 7.2 (Source)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.2.1511 (Core)
CentOS Linux release 7.2.1511 (Core)
cpe:/o:centos:centos:7

[root@Master ~]# uname -r
3.10.0-327.el7.x86_64

Procedure:

Fire up the VMs

$vagrant up

Note: There is an issue when running Kubernetes in vagrant VM environment. By default kubeadm script picks vagrant NAT interface (eth0:IP 10.0.2.15) but we need it to pick second interface (eth1) on which Master and Node communicates. In order to force kubeadm to pick eth1  interface edit your  /etc/hosts file so hostname -i returns VM IP address


[root@Master ~]# cat /etc/hosts
192.168.11.11 Master
[root@Master ~]# hostname -i
192.168.11.11
[root@Master ~]#

Try these steps on all VMs (Master and Nodes). I am following installation instruction from official Kubernetes site. It uses kubeadm to install Kubernetes.


//create file kubernetes.repo in this directory /etc/yum.repos.d
[root@Master ~]# cat /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
       https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

//disable SELinux
[root@Master ~]# sudo setenforce 0

[root@Master ~]# sudo yum install -y docker kubeadm kubelet kubectl kubernetes-cni
[root@Master ~]# sudo systemctl enable docker && systemctl start docker
[root@Master ~]# sudo systemctl enable kubelet & systemctl start kubelet

Initialize Master

Try below step on Master only. This command will initialize master. You can allow kubeadm to pick IP address or specify it explicitly which I am doing here. This is the IP address of my Master machine’s eth1 interface. Make sure Nodes can reach Master on this address

At the end of this command it will provide join command for Nodes


//this command may take couple of minutes
[root@Master ~]#  sudo kubeadm init --api-advertise-addresses 192.168.11.11
[kubeadm] WARNING: kubeadm is in alpha, please do not use it for production clusters.
[preflight] Running pre-flight checks
[init] Using Kubernetes version: v1.5.3
[tokens] Generated token: "084173.692e29a481ef443d"
[certificates] Generated Certificate Authority key and certificate.
[certificates] Generated API Server key and certificate
[certificates] Generated Service Account signing keys
[certificates] Created keys and certificates in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 51.082637 seconds
[apiclient] Waiting for at least one node to register and become ready
[apiclient] First node is ready after 1.017582 seconds
[apiclient] Creating a test deployment
[apiclient] Test deployment succeeded
[token-discovery] Created the kube-discovery deployment, waiting for it to become ready
[token-discovery] kube-discovery is ready after 30.503718 seconds
[addons] Created essential addon: kube-proxy
[addons] Created essential addon: kube-dns

Your Kubernetes master has initialized successfully!

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
    http://kubernetes.io/docs/admin/addons/

You can now join any number of machines by running the following on each node:

kubeadm join --token=084173.692e29a481ef443d 192.168.11.11

//keep note of kubeadm join from above command, you need to run this command on
Nodes to join Master
kubeadm join --token=084173.692e29a481ef443d 192.168.11.11

Deploy POD network

Try this command only on master.  Note: As per Kubernetes installation instructions this step needs to be performed before Node join


[root@Master ~]# kubectl apply -f https://git.io/weave-kube

Once a pod network has been installed, you can confirm that it is working by checking that the kube-dns pod is Running in the output of kubectl get pods --all-namespaces.

And once the kube-dns pod is up and running, you can continue by joining your nodes

Join the Master

Try below command on both Nodes to join the Master. This command will start kubelet in Nodes


[root@Node1 ~]# kubeadm join --token=084173.692e29a481ef443d 192.168.11.11
[kubeadm] WARNING: kubeadm is in alpha, please do not use it for production clusters.
[preflight] Running pre-flight checks
[preflight] Starting the kubelet service
[tokens] Validating provided token
[discovery] Created cluster info discovery client, requesting info from "http://192.254.211.168:9898/cluster-info/v1/?token-id=084173"
[discovery] Cluster info object received, verifying signature using given token
[discovery] Cluster info signature and contents are valid, will use API endpoints [https://192.254.211.168:6443]
[bootstrap] Trying to connect to endpoint https://192.168.11.11:6443
[bootstrap] Detected server version: v1.5.3
[bootstrap] Successfully established connection with endpoint "https://192.168.11.11:6443"
[csr] Created API client to obtain unique certificate for this node, generating keys and certificate signing request
[csr] Received signed certificate from the API server:
Issuer: CN=kubernetes | Subject: CN=system:node:Minion_1 | CA: false
Not before: 2017-02-15 22:24:00 +0000 UTC Not After: 2018-02-15 22:24:00 +0000 UTC
[csr] Generating kubelet configuration
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"

Node join complete:
* Certificate signing request sent to master and response
  received.
* Kubelet informed of new secure connection details.

Run 'kubectl get nodes' on the master to see this machine join.

Let’s check what Kubernetes processes have started on Master


//these are the kubernetes related processes are running on Master
[root@Master ~]# netstat -pan
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      21274/etcd
tcp        0      0 127.0.0.1:10251         0.0.0.0:*               LISTEN      21091/kube-schedule
tcp        0      0 127.0.0.1:10252         0.0.0.0:*               LISTEN      21540/kube-controll
tcp        0      0 127.0.0.1:2380          0.0.0.0:*               LISTEN      21274/etcd
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      21406/kube-apiserve
tcp        0      0 0.0.0.0:6783            0.0.0.0:*               LISTEN      4820/weaver
tcp        0      0 127.0.0.1:6784          0.0.0.0:*               LISTEN      4820/weaver
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      20432/kubelet

//all kube-system pods are running which is a good sign
[root@Master ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                              READY     STATUS    RESTARTS   AGE
kube-system   dummy-2088944543-hp7nl            1/1       Running   0          1d
kube-system   etcd-master                       1/1       Running   0          1d
kube-system   kube-apiserver-master             1/1       Running   0          1d
kube-system   kube-controller-manager-master    1/1       Running   0          1d
kube-system   kube-discovery-1769846148-qtjkn   1/1       Running   0          1d
kube-system   kube-dns-2924299975-15b4q         4/4       Running   0          1d
kube-system   kube-proxy-9rfxv                  1/1       Running   0          1d
kube-system   kube-proxy-qh191                  1/1       Running   0          1d
kube-system   kube-proxy-zhtlg                  1/1       Running   0          1d
kube-system   kube-scheduler-master             1/1       Running   0          1d
kube-system   weave-net-bc9k9                   2/2       Running   11         1d
kube-system   weave-net-nx7t0                   2/2       Running   2          1d
kube-system   weave-net-ql04q                   2/2       Running   11         1d
[root@Master ~]#

As you can see Master and Nodes are in ready state. Cluster is ready to deploy pods


[root@Master ~]# kubectl get nodes
NAME      STATUS         AGE
master    Ready,master   1h
node1     Ready          1h
node2     Ready          1h
[root@Master ~]# kubectl cluster-info
Kubernetes master is running at http://Master:8080
KubeDNS is running at http://Master:8080/api/v1/proxy/namespaces/kube-system/services/kube-dns

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

//check components status. everything looks healthy here
[root@Master ~]# kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   {"health": "true"}

Kubernetes version

[root@Master ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1", GitCommit:"82450d03cb057bab0950214ef122b67c83fb11df", GitTreeState:"clean", BuildDate:"2016-12-14T00:57:05Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.3", GitCommit:"029c3a408176b55c30846f0faedf56aae5992e9b", GitTreeState:"clean", BuildDate:"2017-02-15T06:34:56Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
[root@Master ~]#

Kubernetes UI

As I said earlier there are 3 ways to interact with your cluster. Let’s try UI interface I am following procedure specified here

Try below command to check if Kubernetes dashboard already installed in Master

[root@Master ~]# kubectl get pods --all-namespaces | grep dashboard

If it is not installed as in my case try below command to install Kubernetes dashboard

[root@Master ~]# kubectl create -f https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml
deployment "kubernetes-dashboard" created
service "kubernetes-dashboard" created

Try below to setup proxy .kubectl will handle authentication with apiserver and make Dashboard available at http://localhost:8001/ui.

[root@Master ~]# kubectl proxy &
Starting to serve on 127.0.0.1:8001

Open a browser and point it to http://localhost.8001/ui. You should get Kubernetes dashboard UI like this. You can check your cluster status, deploy pod in cluster using ui

kubernetes_3

Deploy pod

Let’s deploy pod using kubectl cli. I am using yaml template. Create below template in your Master. My template file name is single_container_pod.yaml

This template will deploy a pod with one container, in this case a nginx server. I named my pod web-server and exposed container port 8000

[root@Master]# kubectl create -f single_conatiner_pod.yaml


[root@Master ~]# cat single_container_pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: web-server
  labels:
    app: web-server
spec:
  containers:
    - name: nginx
      image: nginx
      ports:
        - containerPort: 8000

Create pod using above template

[root@Master ~]# kubectl create -f single_container_pord.yaml
pod "web-server" created

//1/1 mean this pod is running with one container
[root@Master ~]# kubectl get pods
NAME         READY     STATUS    RESTARTS   AGE
web-server   1/1       Running   0          47s

[root@Master ~]# kubectl describe pod web-server
Name:           web-server
Namespace:      default
Node:           node2/192.168.11.13
Start Time:     Sun, 26 Feb 2017 06:29:28 +0000
Labels:         app=web-server
Status:         Running
IP:             10.36.0.1
Controllers:    <none>
Containers:
  nginx:
    Container ID:       docker://3b63cab5804d1842659c6424369e6b4a163b482f560ed6324460ea16fdce791e
    Image:              nginx
    Image ID:           docker-pullable://docker.io/nginx@sha256:4296639ebdf92f035abf95fee1330449e65990223c899838283c9844b1aaac4c
    Port:               8000/TCP
    State:              Running
      Started:          Sun, 26 Feb 2017 06:29:30 +0000
    Ready:              True
    Restart Count:      0
    Volume Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-pdsm6 (ro)
    Environment Variables:      <none>
Conditions:
  Type          Status
  Initialized   True
  Ready         True
  PodScheduled  True
Volumes:
  default-token-pdsm6:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-pdsm6
QoS Class:      BestEffort
Tolerations:    <none>
Events:
  FirstSeen     LastSeen        Count   From                    SubObjectPath           Type            Reason          Message
  ---------     --------        -----   ----                    -------------           --------        ------          -------
  21s           21s             1       {default-scheduler }                            Normal          Scheduled       Successfully assigned web-server to node2
  20s           20s             1       {kubelet node2}         spec.containers{nginx}  Normal          Pulling         pulling image "nginx"
  19s           19s             1       {kubelet node2}         spec.containers{nginx}  Normal          Pulled          Successfully pulled image "nginx"
  19s           19s             1       {kubelet node2}         spec.containers{nginx}  Normal          Created         Created container with docker id 3b63cab5804d; Security:[seccomp=unconfined]
  19s           19s             1       {kubelet node2}         spec.containers{nginx}  Normal          Started         Started container with docker id 3b63cab5804d

//this command tells you on which Node pod is running. looks like our pod scheduled
in Node2
[root@Master ~]# kubectl get pods -o wide
NAME         READY     STATUS    RESTARTS   AGE       IP          NODE
web-server   1/1       Running   0          2m        10.36.0.1   node2
[root@Master ~]#

You can login to container using kubectl exec command

[root@Master ~]# kubectl exec web-server -it sh
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
15: eth0@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue state UP group default
    link/ether ee:9d:a1:cb:db:ee brd ff:ff:ff:ff:ff:ff
    inet 10.44.0.2/12 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::ec9d:a1ff:fecb:dbee/64 scope link tentative dadfailed
       valid_lft forever preferred_lft forever
# env
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT=tcp://10.96.0.1:443
HOSTNAME=web-server
HOME=/root
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
NGINX_VERSION=1.11.10-1~jessie
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_SERVICE_HOST=10.96.0.1
PWD=/
#

Let’s login to Node2 and check container using Docker cli

//as can be seen nginx container is running in Node2
[root@Node2 ~]# docker ps
CONTAINER ID        IMAGE                                              COMMAND                  CREATED             STATUS              PORTS               NAMES
3b63cab5804d        nginx                                              "nginx -g 'daemon off"   4 minutes ago       Up 4 minutes                            k8s_nginx.f4302f56_web-server_default_ec8bd607-fbec-11e6-ac27-525400225b53_2dc5c9e9
ce8cd44bd08e        gcr.io/google_containers/pause-amd64:3.0           "/pause"                 4 minutes ago       Up 4 minutes                            k8s_POD.d8dbe16c_web-server_default_ec8bd607-fbec-11e6-ac27-525400225b53_85ec4303
5a20ca6bed11        weaveworks/weave-kube:1.9.0                        "/home/weave/launch.s"   24 hours ago        Up 24 hours                             k8s_weave.c980d315_weave-net-ql04q_kube-system_5f5b0916-fb1e-11e6-ac27-525400225b53_125b8d34
733d2927383f        weaveworks/weave-npc:1.9.0                         "/usr/bin/weave-npc"     24 hours ago        Up 24 hours                             k8s_weave-npc.a8b5954e_weave-net-ql04q_kube-system_5f5b0916-fb1e-11e6-ac27-525400225b53_7ab4a6a7
d270cb27e576        gcr.io/google_containers/kube-proxy-amd64:v1.5.3   "kube-proxy --kubecon"   24 hours ago        Up 24 hours                             k8s_kube-proxy.3cceb559_kube-proxy-zhtlg_kube-system_5f5b707f-fb1e-11e6-ac27-525400225b53_b38dc39e
042abc6ec49c        gcr.io/google_containers/pause-amd64:3.0           "/pause"                 24 hours ago        Up 24 hours                             k8s_POD.d8dbe16c_weave-net-ql04q_kube-system_5f5b0916-fb1e-11e6-ac27-525400225b53_02af8f33
56d00c47759f        gcr.io/google_containers/pause-amd64:3.0           "/pause"                 24 hours ago        Up 24 hours                             k8s_POD.d8dbe16c_kube-proxy-zhtlg_kube-system_5f5b707f-fb1e-11e6-ac27-525400225b53_56485a90

//docker nginx images loaded in Node2
[root@Node2 ~]# docker images
REPOSITORY                                  TAG                 IMAGE ID            CREATED             SIZE
docker.io/nginx                             latest              db079554b4d2        10 days ago         181.8 MB
gcr.io/google_containers/kube-proxy-amd64   v1.5.3              932ee3606ada        10 days ago         173.5 MB
docker.io/weaveworks/weave-npc              1.9.0               460b9ad16e86        3 weeks ago         58.22 MB
docker.io/weaveworks/weave-kube             1.9.0               568b0ac069ad        3 weeks ago         162.7 MB
gcr.io/google_containers/pause-amd64        3.0                 99e59f495ffa        9 months ago        746.9 kB

Delete  pod, try these commands on Master

[root@Master ~]# kubectl get pods
NAME         READY     STATUS    RESTARTS   AGE
web-server   1/1       Running   0          59m

[root@Master ~]# kubectl delete pod web-server
pod "web-server" deleted

[root@Master ~]# kubectl get pods
No resources found.

Check Node2 and  make sure container is deleted

//as you can see there is no nginx container running on Node2
[root@Node2 ~]# docker ps
CONTAINER ID        IMAGE                                              COMMAND                  CREATED             STATUS              PORTS               NAMES
5a20ca6bed11        weaveworks/weave-kube:1.9.0                        "/home/weave/launch.s"   24 hours ago        Up 24 hours                             k8s_weave.c980d315_weave-net-ql04q_kube-system_5f5b0916-fb1e-11e6-ac27-525400225b53_125b8d34
733d2927383f        weaveworks/weave-npc:1.9.0                         "/usr/bin/weave-npc"     24 hours ago        Up 24 hours                             k8s_weave-npc.a8b5954e_weave-net-ql04q_kube-system_5f5b0916-fb1e-11e6-ac27-525400225b53_7ab4a6a7
d270cb27e576        gcr.io/google_containers/kube-proxy-amd64:v1.5.3   "kube-proxy --kubecon"   24 hours ago        Up 24 hours                             k8s_kube-proxy.3cceb559_kube-proxy-zhtlg_kube-system_5f5b707f-fb1e-11e6-ac27-525400225b53_b38dc39e
042abc6ec49c        gcr.io/google_containers/pause-amd64:3.0           "/pause"                 24 hours ago        Up 24 hours                             k8s_POD.d8dbe16c_weave-net-ql04q_kube-system_5f5b0916-fb1e-11e6-ac27-525400225b53_02af8f33
56d00c47759f        gcr.io/google_containers/pause-amd64:3.0           "/pause"                 24 hours ago        Up 24 hours                             k8s_POD.d8dbe16c_kube-proxy-zhtlg_kube-system_5f5b707f-fb1e-11e6-ac27-525400225b53_56485a90

//image remains for future use
[root@Node2 ~]# docker images
REPOSITORY                                  TAG                 IMAGE ID            CREATED             SIZE
docker.io/nginx                             latest              db079554b4d2        10 days ago         181.8 MB
gcr.io/google_containers/kube-proxy-amd64   v1.5.3              932ee3606ada        10 days ago         173.5 MB
docker.io/weaveworks/weave-npc              1.9.0               460b9ad16e86        3 weeks ago         58.22 MB
docker.io/weaveworks/weave-kube             1.9.0               568b0ac069ad        3 weeks ago         162.7 MB
gcr.io/google_containers/pause-amd64        3.0                 99e59f495ffa        9 months ago        746.9 kB
[root@Node2 ~]#

Replication controller

Create yaml template for replication controller. You can read more about replication controller here.

This template replicating 10 pods using ‘replicas:10’.

[root@Master ~]# cat web-rc.yaml
apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx
spec:
  replicas: 10
  selector:
    app: web-server
  template:
    metadata:
      name: web-server
      labels:
        app: web-server
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 8000

Execute replication controller


[root@Master ~]# kubectl create -f web-rc.yaml
replicationcontroller "nginx" created

//10 pods created
[root@Master ~]# kubectl get pods
NAME          READY     STATUS    RESTARTS   AGE
nginx-498mx   1/1       Running   0          10s
nginx-9vfsd   1/1       Running   0          10s
nginx-dgvg6   1/1       Running   0          10s
nginx-fh4bv   1/1       Running   0          10s
nginx-k7j9d   1/1       Running   0          10s
nginx-mz5r0   1/1       Running   0          10s
nginx-q2z79   1/1       Running   0          10s
nginx-w6b4d   1/1       Running   0          10s
nginx-wkshq   1/1       Running   0          10s
nginx-wz7ss   1/1       Running   0          10s


[root@Master ~]# kubectl describe replicationcontrollers/nginx
Name:           nginx
Namespace:      default
Image(s):       nginx
Selector:       app=web-server
Labels:         app=web-server
Replicas:       10 current / 10 desired
Pods Status:    10 Running / 0 Waiting / 0 Succeeded / 0 Failed
No volumes.
Events:
  FirstSeen     LastSeen        Count   From                            SubObjectPath   Type            Reason                  Message
  ---------     --------        -----   ----                            -------------   --------        ------                  -------
  2m            2m              1       {replication-controller }                       Normal          SuccessfulCreate        Created pod: nginx-fh4bv
  2m            2m              1       {replication-controller }                       Normal          SuccessfulCreate        Created pod: nginx-k7j9d
  2m            2m              1       {replication-controller }                       Normal          SuccessfulCreate        Created pod: nginx-mz5r0
  2m            2m              1       {replication-controller }                       Normal          SuccessfulCreate        Created pod: nginx-dgvg6
  2m            2m              1       {replication-controller }                       Normal          SuccessfulCreate        Created pod: nginx-498mx
  2m            2m              1       {replication-controller }                       Normal          SuccessfulCreate        Created pod: nginx-w6b4d
  2m            2m              1       {replication-controller }                       Normal          SuccessfulCreate        Created pod: nginx-9vfsd
  2m            2m              1       {replication-controller }                       Normal          SuccessfulCreate        Created pod: nginx-q2z79
  2m            2m              1       {replication-controller }                       Normal          SuccessfulCreate        Created pod: nginx-wkshq
  2m            2m              1       {replication-controller }                       Normal          SuccessfulCreate        (events with common reason combined)
[root@Master ~]#

Delete one pod. Since we desire 10 replicas, replication controller will restart another pod so total pods are always 10


[root@Master ~]# kubectl get pods
NAME          READY     STATUS    RESTARTS   AGE
nginx-498mx   1/1       Running   0          6m
nginx-9vfsd   1/1       Running   0          6m
nginx-dgvg6   1/1       Running   0          6m
nginx-fh4bv   1/1       Running   0          6m
nginx-k7j9d   1/1       Running   0          6m
nginx-mz5r0   1/1       Running   0          6m
nginx-q2z79   1/1       Running   0          6m
nginx-w6b4d   1/1       Running   0          6m
nginx-wkshq   1/1       Running   0          6m
nginx-wz7ss   1/1       Running   0          6m

[root@Master ~]# kubectl delete pod nginx-k7j9d
pod "nginx-k7j9d" deleted

[root@Master ~]# kubectl get pods
NAME          READY     STATUS              RESTARTS   AGE
nginx-498mx   1/1       Running             0          6m
nginx-74qp9   0/1       ContainerCreating   0          3s
nginx-9vfsd   1/1       Running             0          6m
nginx-dgvg6   1/1       Running             0          6m
nginx-fh4bv   1/1       Running             0          6m
nginx-mz5r0   1/1       Running             0          6m
nginx-q2z79   1/1       Running             0          6m
nginx-w6b4d   1/1       Running             0          6m
nginx-wkshq   1/1       Running             0          6m
nginx-wz7ss   1/1       Running             0          6m

[root@Master ~]# kubectl get pods
NAME          READY     STATUS    RESTARTS   AGE
nginx-498mx   1/1       Running   0          6m
nginx-74qp9   1/1       Running   0          6s
nginx-9vfsd   1/1       Running   0          6m
nginx-dgvg6   1/1       Running   0          6m
nginx-fh4bv   1/1       Running   0          6m
nginx-mz5r0   1/1       Running   0          6m
nginx-q2z79   1/1       Running   0          6m
nginx-w6b4d   1/1       Running   0          6m
nginx-wkshq   1/1       Running   0          6m
nginx-wz7ss   1/1       Running   0          6m
[root@Master ~]#

Increase and decrease number of replicas

[root@Master ~]# kubectl scale --replicas=15 replicationcontroller/nginx
replicationcontroller "nginx" scaled
//increase number of replicas to 15
[root@Master ~]# kubectl get pods
NAME          READY     STATUS    RESTARTS   AGE
nginx-1jdn9   1/1       Running   0          7s
nginx-498mx   1/1       Running   0          17m
nginx-74qp9   1/1       Running   0          11m
nginx-9vfsd   1/1       Running   0          17m
nginx-bgdc6   1/1       Running   0          7s
nginx-dgvg6   1/1       Running   0          17m
nginx-fh4bv   1/1       Running   0          17m
nginx-j2xtf   1/1       Running   0          7s
nginx-m8vlq   1/1       Running   0          7s
nginx-mz5r0   1/1       Running   0          17m
nginx-q2z79   1/1       Running   0          17m
nginx-rmrqt   1/1       Running   0          7s
nginx-w6b4d   1/1       Running   0          17m
nginx-wkshq   1/1       Running   0          17m
nginx-wz7ss   1/1       Running   0          17m
[root@Master ~]#

[root@Master ~]# kubectl scale --replicas=5 replicationcontroller/nginx
replicationcontroller "nginx" scaled
[root@Master ~]# kubectl get pods
NAME          READY     STATUS        RESTARTS   AGE
nginx-9vfsd   1/1       Running       0          19m
nginx-dgvg6   1/1       Running       0          19m
nginx-fh4bv   1/1       Running       0          19m
nginx-mz5r0   0/1       Terminating   0          19m
nginx-q2z79   1/1       Running       0          19m
nginx-w6b4d   1/1       Running       0          19m

[root@Master ~]# kubectl get pods
NAME          READY     STATUS    RESTARTS   AGE
nginx-9vfsd   1/1       Running   0          19m
nginx-dgvg6   1/1       Running   0          19m
nginx-fh4bv   1/1       Running   0          19m
nginx-q2z79   1/1       Running   0          19m
nginx-w6b4d   1/1       Running   0          19m
[root@Master ~]#

Note: I found that after vagrant VMs shutdown and restarts things doesn’t work properly. I see  API server doesn’t come up. Kubernetes documentation explain these steps to restart you cluster if you get into problem. I tried it but only able to bring up cluster with one Node

Reset cluster. Perform below on all VMs


#kubeadm reset

Redo steps


#systemctl enable kubelet && systemctl start kubelet
#kubeadm init --api-advertise-addresses 192.168.11.11
#kubectl apply -f https://git.io/weave-kube
#kubeadm join

Save

Save

Save

Lab-5:Python for network automation – NETCONF client

In this lab I will demonstrate how to create NETCONF connection using ncclient package and Juniper PyEZ NETCONF clients

Component required:

NETCONF capable device. I am using Juniper virtual SRX

Pre-condition:

  • Install ncclient package
    • pip install ncclient
  • Install Juniper PyEZ package
    • pip install junos-eznc
  • Enable NETCONF on device. In my case it is srx
    • configure>edit system services netconf ssh

Procedure:

NETCONF connection using Juniper PyEZ NETCONF client

root@rtxl:/home/sjakhwal/scripts# python

Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type “help”, “copyright”, “credits” or “license” for more information.
>>>
>>> import sys
>>> from jnpr.junos import Device
>>> import json
>>>from jnpr.junos import Device
>>>from lxml import etree
>>>from jnpr.junos.utils.config import Config

>>> dev = Device(host=’192.168.122.35′,user=’root’,passwd=’juniper1′)
>>> dev.open()

Device(192.168.122.35)

>>> print dev.facts

{'domain': None, 'hostname': '', 'ifd_style': 'CLASSIC', 'version_info': junos.version_info(major=(12, 1), type=X, minor=(47, 'D', 20), build=7), '2RE': False, 'serialnumber': 'f9bf7cca7bfa', 'fqdn': '', 'virtual'

>>> dir(dev)

['ON_JUNOS', 'Template', '__class__', '__delattr__', '__dict__', '__doc__', '__enter__', '__exit__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_auth_password', '_auth_user', '_auto_probe', '_conf_auth_user', '_conf_ssh_private_key_file', '_conn', '_facts', '_gather_facts', '_hostname', '_j2ldr', '_manages', '_nc_transform', '_norm_transform', '_normalize', '_port', '_ssh_config', '_ssh_private_key_file', '_sshconf_lkup', '_sshconf_path', 'auto_probe', 'bind', 'cli', 'close', 'connected', 'display_xml_rpc', 'execute', 'facts', 'facts_refresh', 'hostname', 'logfile', 'manages', 'open', 'password', 'probe', 'rpc', 'timeout', 'transform', 'user']

>> print dev.hostname

192.168.122.35

>> print json.dumps(dev.facts)

{"domain": null, "hostname": "", "ifd_style": "CLASSIC", "version_info": {"major": [12, 1], "type": "X", "build": 7, "minor": [47, "D", 20]}, "2RE": false, "serialnumber": "f9bf7cca7bfa", "fqdn": "", "virtual": true, "switch_style": "NONE", "version

>>>print dev.cli(‘show route’)

/usr/local/lib/python2.7/dist-packages/jnpr/junos/device.py:652: RuntimeWarning: CLI command is for debug use only! warnings.warn("CLI command is for debug use only!", RuntimeWarning) inet.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both
0.0.0.0/0         *[Access-internal/12] 6d 23:51:27
> to 192.168.122.1 via ge-0/0/0.0 11.11.11.0/24     
*[Direct/0] 2d 05:49:01 > via ge-0/0/1.0

>>> print dev.display_xml_rpc(‘show route’,format=’text’)

<get-route-information> </get-route-information>

>>>print etree.tostring(dev.rpc.get_route_information({‘format’:’xml’}))

 <route-information>
 <!-- keepalive -->
 <route-table>
 <table-name>inet.0</table-name>
 <destination-count>9</destination-count>
 <total-route-count>9</total-route-count>
 <active-route-count>9</active-route-count>
 <holddown-route-count>0</holddown-route-count>
 <hidden-route-count>0</hidden-route-count>
 <rt style="brief">
 <rt-destination>0.0.0.0/0</rt-destination>
 <rt-entry>
 <active-tag>*</active-tag>
 <current-active/>
 <last-active/>
 <protocol-name>Access-internal</protocol-name>
 <preference>12</preference>
 <age seconds="660641">1w0d 15:30:41</age>
 <nh>
 <selected-next-hop/>
 <to>192.168.122.1</to>
 <via>ge-0/0/0.0</via>
 </nh>
 </rt-entry>
 </rt>
 <rt style="brief">
…..<truncated>…

>>> print dev.display_xml_rpc(‘show interfaces’)

<Element get-interface-information at 0x7fa541ade7a0>

>>> print etree.tostring(dev.rpc.get_interface_information(terse=True))

<interface-information style="terse"> <physical-interface> 
<name> ge-0/0/0 </name> <admin-status> up </admin-status> 
<oper-status>up </oper-status> <logical-interface> 
<name> ge-0/0/0.0
….. <truncated>….

>>>print etree.tostring(dev.rpc.get_interface_information   (interface_name=’ge-0/0/0′))

<interface-information style="normal"> <physical-interface> <name> ge-0/0/0 
</name> <admin-status format="Enabled"> up </admin-status> <oper-status> up 
</oper-status> 
<local-index> 134 </local-index> 
<snmp-index> 507 </snmp-index> 
<link-level-type>


 NETCONF connection using ncclient NETCONF client

Here is another way to create NETCONF connection to your device if it doesn’t provide Python library like Juniper does

sjakhwal@rtxl3rld05:~$ python

Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> from ncclient import manager

>>> dev = manager.connect(host=’192.168.122.35′,username=’root’,password=’juniper1′,hostket_verify=False,timeout=10)

>> dir(dev)

['_Manager__set_async_mode', '_Manager__set_raise_mode', 
'_Manager__set_timeout', '__class__', '__delattr__', '__dict__', '__doc__', 
'__enter__', '__exit__', '__format__', '__getattr__', '__getattribute__', 
'__hash__', '__init__', '__metaclass__', '__module__', '__new__', '__reduce__', 
'__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', 
'__subclasshook__', '__weakref__', '_async_mode', '_device_handler', 
'_raise_mode', '_session', '_timeout', 'async_mode', 'channel_id', 'channel_name', 
'client_capabilities', 'close_session', 'command', 'commit', 
'compare_configuration', 'connected', 'copy_config', 'delete_config', 
'discard_changes', 'dispatch', 'edit_config', 'execute', 'get', 'get_config', 
'get_configuration', 'get_schema', 'halt', 'kill_session', 'load_configuration', 'lock',
'locked', 'poweroff_machine', 'raise_mode', 'reboot', 'reboot_machine', 'rpc', 'scp',
'server_capabilities', 'session', 'session_id', 'timeout', 'unlock', 'validate']

>>> dev.command(command=’show route’,format=’xml’)

<ncclient.xml_.NCElement object at 0x7fdff06d6c90>

>>> print dev.command(command=’show route’,format=’xml’)

<rpc-reply message-id="urn:uuid:896eb964-e7a2-11e5-a0a3-6431501eb7ad">
<route-information>
<!-- keepalive -->
<route-table>
<table-name>inet.0</table-name>
<destination-count>9</destination-count>
<total-route-count>9</total-route-count>
<active-route-count>9</active-route-count>
<holddown-route-count>0</holddown-route-count>
<hidden-route-count>0</hidden-route-count>
<rt style="brief">
<rt-destination>0.0.0.0/0</rt-destination>
<rt-entry>
<active-tag>*</active-tag>
<current-active/>
<last-active/>
<protocol-name>Access-internal</protocol-name>
<preference>12</preference>
<age seconds="748256">1w1d 15:50:56</age>
<nh>
<selected-next-hop/>
<to>192.168.122.1</to>
<via>ge-0/0/0.0</via>
</nh>
</rt-entry>
</rt>

>>> print dev.get_configuration(‘running’)

<rpc-reply message-id="urn:uuid:ddd4e5b4-e7a2-11e5-a0a3-6431501eb7ad">
<configuration changed-seconds="1457375295" changed-localtime="2016-03-07 18:28:15 UTC">
<version>12.1X47-D20.7</version>
<system>
<root-authentication>
<encrypted-password>$1$KhbpR2sm$x9rV5uZSS/Q1a4YRculZ//</encrypted-password>
 </root-authentication>
<login>
<user>
 <name>admin</name>
<uid>2000</uid>
 <class>super-user</class>
<authentication>
<encrypted-password>$1$5QcrZTKt$fqisDA5ZAXQiBOeTN4fH6.</encrypted-password>
</authentication>
</user>
</login>
<services>
<ssh>
</ssh>