How the orchestration engine works

The following section explains how the orchestration engine works in details.

OpenStack Heat vs. Cloud Topology Designer

To understand the orchestration engine, we compare it with the OpenStack HEAT orchestration engine as in Figure 1:

Fig. Cloud Topology Designer vs OpenStack Heat Figure 1

Active connection is not required

The HEAT orchestrator (Figure 1, on the left) requires an agent pre-installed on the VMs. After the VMs start, the agent runs as a daemon process on the VMs. Both the agent and the HEAT orchestrator connect to the same messaging queue (i.e., RabbitMQ). The HEAT orchestrator pushes tasks to the queue. The agent pulls tasks from it and sets up the VMs accordingly. This orchestration type requires an active AMQP connection all the time.

Unlike HEAT, our orchestrator does not require an agent pre-installed and running all the time on the VMs. Instead, it uses Ansible to deploy a service catalog on the target VMs over a bastion host. Before the deployment, the orchestrator creates a security group on the bastion host that allows it to access the bastion host (i.e., allow TCP incoming traffic on port 22 from the remote IP of the orchestrator). After the deployment completes, this security group rule is auto deleted.

Figure 2 shows an example: At the beginning of the workflow, the orchestrator creates a security group on the bastion host (step 1), then enables TCP forwarding on the bastion host (step 2), and auto-deletes the security group rule after the deployment completes (step 3).

Fig. Deployment workflow Figure 2

note

In comparison to HEAT, this orchestrator does not require an active AMQP connection on the VMs all the time.

Fully isolated workflow

For each ansible execution in the workflow, the orchestrator starts a new ansible controller container and executes ansible from inside the container. After a workflow step completes or fails, the orchestrator terminates the container. As a result, a workflow execution from multiple tenants are fully isolated with each other.

Fig. CTD orchestrator starts a sandbox container

Key Management System supported

When users deploy their applications, the orchestration engine uses the user OpenStack token to work on behalf of them and provision resources on Open Telekom Cloud. It means, the orchestration engine itself cannot make any changes to the tenant without the user authentication. This is similar to the OpenStack Heat orchestration engine, which also uses the OpenStack token to provision resource on behalf of the authenticated users.

Recall that the orchestrator uses ansible to deploy service catalogs on the computes via SSH. For this purpose, the orchestrator auto-generates an SSH key for each deployment. The public part of the SSH key is installed on the VMs (via cloud-init). The private part is encrypted in the Key Management System using the user OpenStack token. Without the user OpenStack token, the orchestrator itself cannot decrypt this private key.

Error handling of the orchestration engine

The orchestrator retries a workflow step when the following errors occur:

Terraform error

  • The orchestrator fails to apply terraform while creating (or deleting) an OpenStack compute, router, floating IP, network, port, security group, security group rule.
  • Retry: 1 time.

REST APIs error

The backend server is unreachable

  • The backend server of Open Telekom Cloud is temporarily not available and the REST API responses an HTTP error message The backend server is unreachable.
  • Retry: 5 times.

Error during VM bootstrap

  • The VM is created successfully, but the orchestrator cannot check the connection to the VM after 3 min. Fig. Orchestration engine deploys the HelloWorld python script via SSH

  • The deployment fails immediately. Users should login to the VM and check why cloud-init fails.

Ansible execution connection error

  • Ansible is executed on a VM but terminated with exit code 4 (i.e., unreachable host).
  • Retry: 1 time.

Concurrent workflow steps error

  • When one step in the workflow fails, the orchestrator waits 2 minutes for the other concurrent steps in the workflow to complete. If they do not terminate after 2 minutes, the orchestrator also sets them as failed.