Spinnaker Installation, Deployments, Questions & Answers

A General Review of Spinnaker

The focus of the article will be on end-to-end testing – everything about Spinnaker.

INSTALLATION

Very briefly, on the topic of installing Spinnaker at AWS:
– Follow their documentation
– There are a series of steps regarding IAM Roles covered in https://www.spinnaker.io/setup/providers/aws/ . These are a bit tricky, and if you don’t implement them all of them exactly correctly, Spinnaker will break.
– The VPC should preferably be a “Default VPC”. Check this.
– The VPC should be given a Name tag that doesn’t include a . character (e.g. myvpc)
– The subnets should each be given a Name tag that follows vpcname.subnetpurpose.availability-zone – e.g. myvpc.internal.us-west-2a
– Add a number of AWS Regions, even including (yet again) the default region. Such as
hal config provider aws account edit my-aws-account –add-region us-west-2
hal config provider aws account edit my-aws-account –add-region us-west-1
hal deploy apply
– After launching both Spinnaker and Halyard, make sure there are up to 8 java processes running. They are:
rosco
orca
igor
gate
front50
clouddriver
echo
halyard

Here are ways to check:

ps -ef
ps -ef | grep java | grep -v grep | wc

That makes 8 java processes. If fewer are running, observe the log files:

cd /var/log/spinnaker
ls -alR

Find the log file of the process which has stopped, and observe error messages. Fix the problems and restart spinnaker. Make sure all the processes are running. Later on, it will also be helpful to find errors in these logs.

CONNECTING TO THE SPINNAKER SERVER

Spinnaker only listens on the localhost by default.
Earlier official documentation had discussed tunneling, however it may have been removed. Here are the steps. On your dev workstation,

~/.ssh/config :

Host spinnaker-start
   HostName 1.2.3.4
   IdentityFile /path/to/my-aws-account-keypair.pem
   ControlMaster yes
   ControlPath ~/.ssh/spinnaker-tunnel.ctl
   RequestTTY no
   LocalForward 9000 127.0.0.1:9000
   LocalForward 8084 127.0.0.1:8084
   LocalForward 8087 127.0.0.1:8087
   User ubuntu

Host spinnaker-stop
   HostName 1.2.3.4
   IdentityFile /path/to/my-aws-account-keypair.pem
   ControlPath ~/.ssh/spinnaker-tunnel.ctl
   RequestTTY no

Host spinnaker
   HostName 1.2.3.4
   IdentityFile /path/to/my-aws-account-keypair.pem
   User ubuntu

replace HostName and IdentityFile.

spinnaker-tunnel.sh :

#!/usr/bin/bash

socket=$HOME/.ssh/spinnaker-tunnel.ctl

if [ "$1" == "" ]; then
    echo "Please run either ./spinnaker-tunnel.sh start or ./spinnaker-tunnel.sh stop"
fi

if [ "$1" == "start" ]; then
   if [ ! \( -e ${socket} \) ]; then
     echo "Starting tunnel to Spinnaker..."
     ssh -f -N spinnaker-start && echo "Done."
   else
     echo "Tunnel to Spinnaker running."
   fi
fi

if [ "$1" == "stop" ]; then
   if [ \( -e ${socket} \) ]; then
     echo "Stopping tunnel to Spinnaker..."
     ssh -O "exit" spinnaker-stop && echo "Done."
   else
     echo "Tunnel to Spinnaker stopped."
   fi
fi

chmod 755 spinnaker-tunnel.sh

ssh-start.sh :

#!/bin/bash

eval `ssh-agent`
ssh-add /path/to/my-aws-account-keypair.pem

chmod 755 ssh-start.sh

Then, in order to connect
. ssh-start.sh #only once
./spinnaker-tunnel.sh start #whenever the tunnel should be started
./spinnaker-tunnel.sh stop #whenever the tunnel should be stopped

In order to customize the configurations of the various services, there are many config files such as

/opt/spinnaker/config/spinnaker.yml	
/opt/rosco/config/rosco.yml

and many more. I have written another post dedicated only to understanding the config files: Spinnaker Configuration File Precedence

DEPLOYMENTS

Once you have Spinnaker open in a browser, the basic steps are:
– Create an app
– Create a pipeline
– Add “bake” and “deploy” stages for the pipeline
– Run this, and debug the problems
– Observe the clusters which are generated
– Try destroying and recreating clusters with Spinnaker

Rather than walk through all those steps in this article, which would perhaps be boring, another strategy is to say: just go ahead and try those steps, on your own. You may run into problems or questions. Let’s try to answer those now.

Questions and Answers:

Q: The bake stage has many options, and it’s generally confusing. How is that set up?
A: See Spinnaker, Ansible and Packer

Q: If you bake in multiple regions, how do you deploy in multiple regions? one stage? multiple stages?
A: For the “Deploy Configuration”, you can add multiple server groups. They each are assigned the correct AMI from the bake stage. This magically “just works”. It keeps track of the AMI’s.

Q: The restriction of only using Ubuntu Trusty instead of Ubuntu Xenial (as of 2017-10 and earlier), is that for Spinnaker server itself, or target images, or both?
A: Just the Spinnaker server itself. The target image may be Xenial.

Q: How can I add more images, such as new versions of Ubuntu, to the deployment pipeline?
A: Here is an example. Modify as necessary.

vi /home/ubuntu/.hal/default/profiles/rosco/rosco.yml

    - baseImage:
        id: xenial
        shortDescription: v16.04
        detailedDescription: Ubuntu Xenial Xerus v16.04
        packageType: deb
      virtualizationSettings:
      - region: eu-west-2
        virtualizationType: hvm
        instanceType: t2.micro
        sourceAmi: ami-996372fd
        sshUserName: ubuntu
      - region: ap-northeast-2
        virtualizationType: hvm
        instanceType: t2.micro
        sourceAmi: ami-d28a53bc
        sshUserName: ubuntu
      - region: ap-south-1
        virtualizationType: hvm
        instanceType: t2.micro
        sourceAmi: ami-099fe766
        sshUserName: ubuntu
hal deploy apply
hal config provider aws account edit my-aws-account --add-region eu-west-2
hal config provider aws account edit my-aws-account --add-region ap-northeast-2
hal config provider aws account edit my-aws-account --remove-region ap-northeast-1
hal config provider aws account edit my-aws-account --add-region ap-south-1
hal deploy apply

Q: Do you have an example app that I could deploy with spinnaker.
A: Yes, https://github.com/sdarwin/ansible-samplewebsite . That is covered in Spinnaker, Ansible and Packer

Q: Why aren’t spinnaker services starting on boot, or restarting on reboot?
A: For Ubuntu 14.04, create and run this script to solve the problem.

# upstarter.sh

#!/bin/bash
for i in clouddriver echo front50 gate igor orca rosco
  do
    echo "manual" > /etc/init/$i.override
  done

chmod 755 upstarter.sh
./upstarter.sh

Q: I am having difficulties setting up email notifications.
A: Instead of Gmail, try Amazon SES, or another dedicated mail relay service. Or, a local installation of Exim or Postfix. The following steps will send mail via the local machine.

vi /home/ubuntu/.hal/default/profiles/echo-local.yml

management.health.mail.enabled: true

mail:
  enabled: true
  from: [email protected]
  host: localhost
  fromAddress: [email protected]
spring:
  mail:
    host: localhost

hal deploy apply

Install Postfix or Exim on the server itself. Check that is working. Send mail from the command-line:

mail [email protected]

Q: The Spinnaker auth configuration looks complicated.
A: Yes. Try SSH tunnels instead, explained above.

Q: What is the secret of spinnaker?
A: Here is an observation about how pipelines work:
One apparently implicit and important fact, is that a generated image in one stage will be sent to the next stage for deployment.
– Bake an image
– Deploy an image
You don’t have to reference it. The pipeline understands it should be the image from the last stage. This would then explain the mysterious “Find an Image”, or “Tag an Image”, which are optional stages. You may think something like “which one? which image?” It must be this implicit image, which is getting passed on from stage to stage in the pipeline.

Q: How do I upgrade Hal and Spinnaker?
A: Example:

sudo apt-get update && sudo apt-get upgrade spinnaker-halyard
hal version list
hal config version edit --version 1.4.2
hal deploy apply 

Q: How do you add a scaling policy for an autoscaling group?
A: After deploying clusters in Spinnaker, add the scaling policy in AWS, rather than the Spinnaker interface.

Q: Must you recreate such scaling policies every time, in AWS?
A: Subsequent deployments of the same pipeline will use the previous scaling policy.

Q: If something changes in AWS, does Spinnaker pick it up? Is there a two-way street between AWS configs and Spinnaker configs?
A: Yes

Q: How do min, max, and desired levels work for autoscaling groups?
A: The “desired” level will fluctuate during autoscaling. If the instances are under load, this setting increases, up until the “max” level. If the instances are not under load, this setting will decrease, down to the “min” level. You should configure “min” and “max”, and let autoscaling do the rest.

Q: What does disabling a cluster accomplish?
A: This removes the instances from the Load Balancer, without deleting them.

Leave a Reply

Your email address will not be published. Required fields are marked *