Guides

• 1: Adjutant
• 2: Affinity Policy
• 3: API access
• 4: Application credentials
• 5: Barbican
• 6: Billing
• 7: Detach & Attach interface on a Ubuntu instance
• 8: EC2 Credentials
• 9: Getting started with OpenStack
• 10: Octavia
• 11: Recover root password or failed mount
• 12: Required xfs mount options
• 13: Swift ACL
• 14: Swift getting started
• 15: Swift S3 compatibility
• 16: Terraform Backend
• 17: Volume Attachment Limits
• 18: Volume Backup & Restore
• 19: Volume migration
• 20: Volume Retype
• 21: Windows volume offline after restart

1 - Adjutant

Overview

OpenStack Adjutant is a service that allows users to manage projects and their users directly from Horizon.

User management

Users can be managed directly from the management tab in Horizon dashboard.

To invite a new member to your project(s) go to “Management” -> “Access Control” -> “Project Users” in the menu, and then click “Invite User” on the right hand side. A popup will appear with a textbox where you can type the email of the new user. Below the textbox you can chose which roles (described below) to assign that user. Click “Invite” when done and an invite will be sent via email.

Note: If the user doesn’t have an OpenStack account with us already, they will have to follow the email instructions and sign up.

There are a couple of roles that can be assigned to users inside of a project:

• Load Balancer - Allow access to manage load balancers (Octavia).
• Object Store - Allow access to manage objects in object store (Swift).
• Orchestration - Allow access to manage orchestration templates (Heat).
• Project Administrator - Full control over the project, including adding and removing other project administrators.
• Project Member - Allow access to core services such as compute (Nova), network (Neutron) and volume (Cinder).
• Project Moderator - Can invite and manage project members, but not project administrators.
• Secret Store - Allow access to manage objects inside of secret store (Barbican).

2 - Affinity Policy

Overview

Here is how to avoid that groups of instances run on the same compute node. This can be relevant when configuring resilience.

1. Create an anti affinity group.
   Take note of the group UUID that is displayed when created. It is needed when deploying the instance.
   openstack server group create --policy anti-affinity testgroup

   https://docs.openstack.org/python-openstackclient/train/cli/command-objects/server-group.html

2. (Optional) Read out the affinity policies.
   openstack server group list | grep -Ei "Policies|affinity"

3. Add the instance to the group when deploying.
   openstack server create --image ubuntu-20.04-server-latest --flavor v1-small-1 --hint group=<server_group_uuid> test-instance

   https://docs.openstack.org/python-openstackclient/train/cli/command-objects/server.html

Additional links

https://docs.openstack.org/senlin/train/user/policy_types/affinity.html

3 - API access

Introduction

OpenStack provides REST APIs for programmatic interaction with the various services (compute, object storage, etc.). These APIs are used by automation tools such as HashiCorp Terraform and the OpenStack CLI utility.

For advanced programmatic usage, there exist freely available SDKs and software libraries for several languages which are maintained by the OpenStack project or community members.

This guides describes the initial steps required for manual usage of the OpenStack REST APIs.

Authentication

Usage of an application credential for API authentication is recommend due to their security and operational benefits.

Listing endpoints

API endpoints for the OpenStack services can be listed by navigating to “Project” → “API Access” in the Horizon web console or by issuing the following command:

$ openstack catalog list

Endpoints marked as “public” in the command output are intended for customer usage.

Reference documentation

For detailed usage of the of the APIs, see the official OpenStack API reference documentation (version “Train”).

4 - Application credentials

Overview

This guide will help you getting started with OpenStack application credentials. Application credentials are designed to be used by automation and CLI tools, such as Terraform and the OpenStack command-line client.

Create application credential using web dashboard

Navigate to “Identity” → “Application Credentials” in your target project and press “Create Application Credential”. Once created, you’ll be offered to download the generated credential configuration as an OpenStack RC file (“openrc” version 3) or in the “clouds.yaml” format.

Create application credential using CLI

To create a pair of application credentials run the openstack application credential create <name> command. By default the same access as the user running the command will be given. If you wish to override the roles given add --role <role> for each role you want to add.

You can also set an expiration date when creating a pair of application credentials, add the flag --expiration followed by a timestamp in the following format: YYYY-mm-ddTHH:MM:SS.

For more detail you can visit the OpenStack documentation that goes more into detail on all avaible options.

An example that will give access to the most commonly used APIs:

openstack application credential create test --role _member_ --role creator --role load-balancer_member

+--------------+----------------------------------------------------------------------------------------+
| Field        | Value                                                                                  |
+--------------+----------------------------------------------------------------------------------------+
| description  | None                                                                                   |
| expires_at   | None                                                                                   |
| id           | 3cd933bbcf824bdc9f77f37692eea60a                                                       |
| name         | test                                                                                   |
| project_id   | bb301d6172f54d749f9aa3094d77eeef                                                       |
| roles        | _member_ creator load-balancer_member                                                  |
| secret       | ibHyYuIPQCf-IKVN0qOEAgf4CNvDWmT5ltI6mdbmUTMD7OvJTu-5nXX0U6_5EOXTKriq7C7Ka06wKmJa0yLcKg |
| unrestricted | False                                                                                  |
+--------------+----------------------------------------------------------------------------------------+

Beware: You will not be able to view the secret again after creation. In case you forget the secret you will need to delete and create a new pair of application credentials.

Create an openrc file

#!/usr/bin/env bash
export OS_AUTH_TYPE=v3applicationcredential
export OS_AUTH_URL=https://ops.elastx.cloud:5000/v3
export OS_APPLICATION_CREDENTIAL_ID="<ID>"
export OS_APPLICATION_CREDENTIAL_SECRET="<SECRET>"
export OS_REGION_NAME="se-sto"
export OS_INTERFACE=public
export OS_IDENTITY_API_VERSION=3

Available roles

Below you will find a table with available roles and what they mean.

List application credentials using CLI

To list all existing application credentials available in your project you can run the openstack application credential list command.

Example:

openstack application credential list

+----------------------------------+------+----------------------------------+-------------+------------+
| ID                               | Name | Project ID                       | Description | Expires At |
+----------------------------------+------+----------------------------------+-------------+------------+
| 3cd933bbcf824bdc9f77f37692eea60a | test | bb301d6172f54d749f9aa3094d77eeef | None        | None       |
+----------------------------------+------+----------------------------------+-------------+------------+

Show application credential permissions using CLI

To show which permissions a set of application credentials have you can run the openstack application credential show command followed by the ID of the credential you want to inspect.

Example:

openstack application credential show 3cd933bbcf824bdc9f77f37692eea60a

+--------------+------------------------------------------------------------------------------------+
| Field        | Value                                                                              |
+--------------+------------------------------------------------------------------------------------+
| description  | None                                                                               |
| expires_at   | None                                                                               |
| id           | 3cd933bbcf824bdc9f77f37692eea60a                                                   |
| name         | test                                                                               |
| project_id   | bb301d6172f54d749f9aa3094d77eeef                                                   |
| roles        | creator load-balancer_member _member_                                              |
| unrestricted | False                                                                              |
+--------------+------------------------------------------------------------------------------------+

Delete application credentials using CLI

To delete a pair of application credentials enter the openstack application credential delete command followed by the ID of the credentials you want to remove.

Example:

openstack application credential delete 3cd933bbcf824bdc9f77f37692eea60a

5 - Barbican

Overview

OpenStack Barbican is a key management service for storing highly sensitive data like private keys for certificates and passwords which needs to be available for applications during runtime.

ELASTX Barbican service is backed by physical HSM appliances to ensure that all data is securely stored.

REST API reference can be found here: https://docs.openstack.org/barbican/train/api/index.html

Secrets in Barbican have a special design with regards to ID, they are always referenced by a “secret href” instead of a UUID! (This will change in a later release!)

Secret types

There are a few types of secrets that are handled by barbican:

• symmetric - Used for storing byte arrays such as keys suitable for symmetric encryption.
• public - Used for storing the public key of an asymmetric keypair.
• private - Used for storing the private key of an asymmetric keypair.
• passphrase - Used for storing plain text passphrases.
• certificate - Used for storing cryptographic certificates such as X.509 certificates.
• opaque - Used for backwards compatibility with previous versions of the API without typed secrets. New applications are encouraged to specify one of the other secret types.

Store and fetch a passphrase using openstack cli

Make sure you have installed the openstack python client and the barbican python client.

Store a passphrase as a secret:

$ openstack secret store --secret-type passphrase --name "test passphrase" --payload 'aVerYSecreTTexT!'
+---------------+-------------------------------------------------------------------------------+
| Field         | Value                                                                         |
+---------------+-------------------------------------------------------------------------------+
| Secret href   | https://ops.elastx.cloud:9311/v1/secrets/d9e88d84-c668-48d9-a051-f0df2e23485b |
| Name          | test passphrase                                                               |
| Created       | None                                                                          |
| Status        | None                                                                          |
| Content types | None                                                                          |
| Algorithm     | aes                                                                           |
| Bit length    | 256                                                                           |
| Secret type   | passphrase                                                                    |
| Mode          | cbc                                                                           |
| Expiration    | None                                                                          |
+---------------+-------------------------------------------------------------------------------+

Get information (only metadata) about the secret

$ openstack secret get https://ops.elastx.cloud:9311/v1/secrets/d9e88d84-c668-48d9-a051-f0df2e23485b
+---------------+-------------------------------------------------------------------------------+
| Field         | Value                                                                         |
+---------------+-------------------------------------------------------------------------------+
| Secret href   | https://ops.elastx.cloud:9311/v1/secrets/d9e88d84-c668-48d9-a051-f0df2e23485b |
| Name          | test passphrase                                                               |
| Created       | 2018-12-18T12:13:34+00:00                                                     |
| Status        | ACTIVE                                                                        |
| Content types | {u'default': u'text/plain'}                                                   |
| Algorithm     | aes                                                                           |
| Bit length    | 256                                                                           |
| Secret type   | passphrase                                                                    |
| Mode          | cbc                                                                           |
| Expiration    | None                                                                          |
+---------------+-------------------------------------------------------------------------------+

Get the actual secret

$ openstack secret get --payload https://ops.elastx.cloud:9311/v1/secrets/d9e88d84-c668-48d9-a051-f0df2e23485b
+---------+------------------+
| Field   | Value            |
+---------+------------------+
| Payload | aVerYSecreTTexT! |
+---------+------------------+

Store and fetch a passphrase using the REST API (curl examples)

First get a keystone authentication token (using openstack token issue for example).

Store a passphrase as a secret:

Note that payloads is always base64 encoded when uploaded!
$ echo 'AnotHeRs3crEtT3xT!' | base64
QW5vdEhlUnMzY3JFdFQzeFQhCg==

$ curl -H "X-Auth-Token: $TOKEN" \
    -H 'Accept: application/json' \
    -H 'Content-Type: application/json' \
    https://ops.elastx.cloud:9311/v1/secrets -d '{
      "name": "Test Passphrase REST",
      "secret_type": "passphrase",
      "payload": "QW5vdEhlUnMzY3JFdFQzeFQhCg==",
      "payload_content_type": "application/octet-stream",
      "payload_content_encoding": "base64",
      "algorithm": "AES",
      "bit_length": 256,
      "mode": "CBC"
  }' | python -m json.tool
{
    "secret_ref": "https://ops.elastx.cloud:9311/v1/secrets/85b2df94-a44b-452b-807b-ddcee83d824b"
}

Get the secret payload

$ curl -H "X-Auth-Token: $TOKEN" \
    -H 'Accept: application/octet-stream' \
    https://ops.elastx.cloud:9311/v1/secrets/85b2df94-a44b-452b-807b-ddcee83d824b/payload
AnotHeRs3crEtT3xT!

6 - Billing

Overview

We use OpenStack CloudKitty for billing purposes and with it’s open API it is possible to get detailed information about the cost of resources.

NOTE: The billing data engine is ALWAYS 4 hours behind so it is only possible to retrieve rating data up until 4 hours ago! This is to ensure that all billing data is in place before calculating costs.

Prerequisites

To fetch data from cloudkitty using the OpenStack CLI it is neccessary to install the openstack python client and the openstack cloudkitty python client.

pip install python-openstackclient python-cloudkittyclient

As of this writing, version 5.2.2 of the openstack client and 4.8.0 of the cloudkitty client is working well.

Known limitations

As cloudkitty stores data for a long time, retrieval of data where the begin timestamp is omitted, or more than 1 month ago, will take a very long time or even timeout. Even fetching a month worth of data will take at least 10 minutes so do have patience when exploring your data with this API.

Fetch summary for last month

To fetch the total summary for the last month:

$ openstack rating summary get -b $(date --date='1 month ago' -Isecond)
+----------------------------------+---------------+-------------+---------------------+---------------------+
| Tenant ID                        | Resource Type | Rate        | Begin Time          | End Time            |
+----------------------------------+---------------+-------------+---------------------+---------------------+
| 17cb6c5e5af8481e8960d8c4f4131b0f | ALL           | 47511.96316 | 2024-07-28T18:57:28 | 2024-09-01T00:00:00 |
+----------------------------------+---------------+-------------+---------------------+---------------------+

Fetch dataframes for specific resource types

It is also possible to fetch data for specific resources or resource types. This is specifically useful for getting costs for a specific resouce like an instance or volume.

Group by resource type (-g flag):

$ openstack rating summary get -b $(date --date='1 day ago' -Isecond) -g res_type
+----------------------------------+--------------------------+-----------+---------------------+---------------------+
| Tenant ID                        | Resource Type            | Rate      | Begin Time          | End Time            |
+----------------------------------+--------------------------+-----------+---------------------+---------------------+
| 17cb6c5e5af8481e8960d8c4f4131b0f | network-traffic-sent     | 0         | 2024-08-27T19:59:04 | 2024-09-01T00:00:00 |
| 17cb6c5e5af8481e8960d8c4f4131b0f | network-traffic-received | 0         | 2024-08-27T19:59:04 | 2024-09-01T00:00:00 |
| 17cb6c5e5af8481e8960d8c4f4131b0f | image.size               | 0.16569   | 2024-08-27T19:59:04 | 2024-09-01T00:00:00 |
| 17cb6c5e5af8481e8960d8c4f4131b0f | snapshot.size            | 0.22      | 2024-08-27T19:59:04 | 2024-09-01T00:00:00 |
| 17cb6c5e5af8481e8960d8c4f4131b0f | storage.objects.size     | 1.43383   | 2024-08-27T19:59:04 | 2024-09-01T00:00:00 |
| 17cb6c5e5af8481e8960d8c4f4131b0f | router                   | 14.96     | 2024-08-27T19:59:04 | 2024-09-01T00:00:00 |
| 17cb6c5e5af8481e8960d8c4f4131b0f | ip.floating              | 21.12     | 2024-08-27T19:59:04 | 2024-09-01T00:00:00 |
| 17cb6c5e5af8481e8960d8c4f4131b0f | volume.size              | 686.90159 | 2024-08-27T19:59:04 | 2024-09-01T00:00:00 |
| 17cb6c5e5af8481e8960d8c4f4131b0f | instance                 | 688.09399 | 2024-08-27T19:59:04 | 2024-09-01T00:00:00 |
+----------------------------------+--------------------------+-----------+---------------------+---------------------+

Filter on resource type (-s flag):

$ openstack rating summary get -b $(date --date='1 day ago' -Isecond) -s instance
+----------------------------------+---------------+-----------+---------------------+---------------------+
| Tenant ID                        | Resource Type | Rate      | Begin Time          | End Time            |
+----------------------------------+---------------+-----------+---------------------+---------------------+
| 17cb6c5e5af8481e8960d8c4f4131b0f | instance      | 688.09399 | 2024-08-28T05:59:49 | 2024-09-01T00:00:00 |
+----------------------------------+---------------+-----------+---------------------+---------------------+

Fetch raw dataframes

Cloudkitty is built on a concept called dataframes which is the actual data rated. Each dataframe contains the rated value for each resource for an hour interval and can be exported as a CSV which can then be used to summarize the totals per resource.

Example config for generating a CSV that contains all relevant information

$ cat tmp/cloudkitty.csv 
# This exact file format must be respected (- column_name: json_path)
# The path is interpreted using jsonpath-rw-ext, see
# https://github.com/sileht/python-jsonpath-rw-ext for syntax reference
- 'Begin': '$.begin'
- 'End': '$.end'
- 'Resource Type': '$.service'
- 'Resource ID': '$.desc.id'
- 'Qty': '$.volume'
- 'Cost': '$.rating'

Get raw dataframes for all instances in the project as a CSV for the last 5 hours.

$ openstack rating dataframes get -b $(date --date='5 hours ago' -Isecond) -r instance -f df-to-csv --format-config-file tmp/cloudkitty.csv 
Begin,End,Resource Type,Resource ID,Qty,Cost        
2024-09-10T09:00:00,2024-09-10T10:00:00,instance,064e8601-8c83-477c-85c4-f40884ad71b9,1,3.36
2024-09-10T09:00:00,2024-09-10T10:00:00,instance,21bcc6e2-416a-48c8-8684-2cfaa806e0e3,1,0.14
2024-09-10T10:00:00,2024-09-10T11:00:00,instance,064e8601-8c83-477c-85c4-f40884ad71b9,1,3.36
2024-09-10T10:00:00,2024-09-10T11:00:00,instance,21bcc6e2-416a-48c8-8684-2cfaa806e0e3,1,0.14
2024-09-10T11:00:00,2024-09-10T12:00:00,instance,064e8601-8c83-477c-85c4-f40884ad71b9,1,3.36
2024-09-10T11:00:00,2024-09-10T12:00:00,instance,21bcc6e2-416a-48c8-8684-2cfaa806e0e3,1,0.14

From this CSV output it is fairly easy to sum up the Cost per instance id to get the detailed cost per instance for an interval

7 - Detach & Attach interface on a Ubuntu instance

Overview

If you need to change interface on a Ubuntu instance, then this is the procedure to use.

1. Run the following command in the instance.

    sudo cloud-init clean
   

2. Shut down the instance

3. Detach / Attach the network interface

4. Start the instance

5. Reassociate Floating IP with the instance

8 - EC2 Credentials

Overview

For using the OpenStack S3 API:s you need to generate an additional set of credentials. These can then be used to store data in the Swift Object store for applications that don’t have native Swift support but do support the S3 interfaces.

NOTE: If the application does support Swift natively, using Swift will provide superior performance and generally a better experience.

Create and fetch credentials using openstack cli

Make sure you have installed the openstack python client.

Generate credentials:

$ openstack ec2 credentials create
+-----------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| Field           | Value                                                                                                                                   |
+-----------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| access          | xxxyyyzzz                                                                                                                               |
| access_token_id | None                                                                                                                                    |
| app_cred_id     | None                                                                                                                                    |
| links           | {'self': 'https://ops.elastx.cloud:5000/v3/users/123/credentials/OS-EC2/456'}                                                           |
| project_id      | 123abc                                                                                                                                  |
| secret          | aaabbbccc123                                                                                                                            |
| trust_id        | None                                                                                                                                    |
| user_id         | efg567                                                                                                                                  |
+-----------------+-----------------------------------------------------------------------------------------------------------------------------------------+

Fetch credentials:

$ openstack ec2 credentials list
+----------------------------------+----------------------------------+----------------------------------+----------------------------------+
| Access                           | Secret                           | Project ID                       | User ID                          |
+----------------------------------+----------------------------------+----------------------------------+----------------------------------+
| xxxyyyzzz                        | aaabbbccc123                     | 123abc                           | efg567                           |
+----------------------------------+----------------------------------+----------------------------------+----------------------------------+

Delete credentials

Make sure you have installed the openstack python client.

Use the access key to refer to the credentials you wish to delete:

$ openstack ec2 credentials delete xxxyyyzzz

9 - Getting started with OpenStack

Here are the initial steps that you need to perform to start you first instance and access it with ssh.
In this example we use an ubuntu image and restrict SSH access.

Create Network

1. Go to “Project” > “Network” > “Networks”
2. Select “Create Network” > set a network name > “Next”
3. Set subnet name and network address (10.0.1.0/24 for example) > “Next”
4. Make sure that “Enable DHCP” is checked.
5. Add “DNS Name Servers” > “Create” (ip of a resolver e.g. dns0.eu, dns4eu, Cloudflare, or other depending on your privacy and policy preferences).

Create Router

1. Go to “Project” > “Network” > “Routers”
2. Select “Create Router” > set a router name and select the public network “elx-public1” > “Create Router”
3. Select the router you just created > “Interfaces” > “Add Interface” > select the subnet you created > “Add Interface”

Now the network is up and ready for you to create the first instance.

Create ssh key

1. Go to “Project” > “Compute” > “Key Pairs”
2. Select “Create Key Pair” > set key pair name > “Create Key Pair”
3. Select Key Type -> “SSH Key”
4. Save the private key

Create Security Group

1. Go to “Project” > “Network” > “Security Groups”
2. Select “Create Security Group” > set a name > “Create Security Group”
3. Select “Manage Rules” on the security group you created"
4. Add IP address under “CIDR” to restrict access > “Add” (e.g. 215.1.1.1/32 to only allow this one IP)
5. Select “Add Rule” > set “Port” 22 > add an IP address under “CIDR” to restrict access > “Add”

Create instance

1. Go to “Project” > “Compute” > “Instances”
2. Select “Launch Instance” > Set instance name > Specify Availability Zone > “Next”
3. Select “Image” in “Select Boot Source” > Select “No” in “Create New Volume”
4. Select image (ubuntu-20.04-server-latest for example) > “Next”
5. Select a flavor (v2-c1-m0.5-d20 for example) > “Next”
6. Your network should already be selected > “Next”
7. You do not need to select any port > “Next”
8. Add the security group you created earlier > “Next”
9. The key pair you created earlier should already be selected.
10. “Launch instance”

Specifying the Availability Zone is important if you plan on using volumes, as these can’t be attached nor migrated across Availability Zones

Add a public IP to the instance

1. Go to “Project” > “Compute” > “Instances” > from the “Actions” menu on the instance you created select “Associate Floating IP”
2. Select the “+” button next to the “IP Address” field
3. Select “Pool” “elx-public1” > “Allocate IP”
4. “Associate”

Log in to your new instance

Use the floating IP and the ssh key you created.

In this example the ssh key pair I created was named mykeypair and the public ip is “1.2.3.4” and the image I used was an Ubuntu image. In this example:

ssh -i mykeypair.pem ubuntu@1.2.3.4

The username is different depending on the Linux flavor you are using but you will always use the keypair and not a password.
This is the generic pattern to login from a Linux client:

ssh -l UserName -i /path/to/my-keypair.pem 1.2.3.4

Default UserName is different depending on distribution:

• When using Heat to deploy instances the user name will be ec2-user instead.

Changing the default username

In most modern distributions it’s also possible to change the default username when creating a server by utilizing cloud-init.
The sample configuration below would change the deafult username to “yourusername”.

#cloud-config
system_info:
  default_user:
    name: yourusername

10 - Octavia

This is an example of a minimal setup that includes a basic HTTP loadbalancer. Here is a short explanation of a minimal (configuration) setup from GUI (Horizon).

1. Network -> Loadbalancer -> Create loadbalancer

2. Load Balancer Details: Subnet: Where your webservers live

3. Listener Details: Select HTTP, port 80.

4. Pool Details: This is your “pool of webservers”. Select Algoritm of preference.

5. Pool members: Select your webservers.

6. Finally, proceed to “Create Loadbalancer”.

Note, the loadbalancer will not show up in the Network Topology graph. This is expected.

Octivia features numerous configuration variations. The full reference of variations and CLI guide can be found here: https://docs.openstack.org/octavia/train/user/guides/basic-cookbook.html

Take note that an additional CLI client is required to access the full potential as described above. https://docs.openstack.org/python-octaviaclient/latest/cli/index.html#loadbalancer

11 - Recover root password or failed mount

Overview

If you need to set the root password on a Centos 7 instance or fix a faulty /etc/fstab for example, then this is the procedure to access the boot disk without a root password.

Modify boot parameters

1. Reboot the instance and press “e” when the grub 2 boot menu is shown.

2. Add the following parameters at the end of the linux16 line:

   rd.break enforcing=0

3. Remove all parameters starting with “console=”

4. Press Ctrl+x to boot

Set password

1. Remount the root filesystem read write.

$ mount -o remount,rw /sysroot

2. You can now edit the root disk files under /sysroot.

3. To set a new password do a change root and set password.

$ chroot /sysroot
$ passwd

4. Exit both shells and the instance will reboot.

$ exit

12 - Required xfs mount options

Overview

We have discovered that using xfs file system without the required mount options will make the mount fail in case of storage platform rebalance, maintenance or failover.

When mounting a xfs file system you must mount it with the options noatime,discard,nobarrier.

Note that some cloud images (CentOS 7 for example) have xfs as the default root file system and if you do not use the latest ELASTX provided images you will have to modify the mount options.

Add the mount options to /etc/fstab and reboot or add the mount options online.

mount -o remount,noatime,discard,nobarrier /mountpoint

13 - Swift ACL

Overview

In this guide we will go through how to manage user permissions for Swift containers using ACLs.

Due to current limitation of how privilege/role management works, the way to go is to create a separate “Swift project”.

The purpose of so called “Swift projects” ("_swift") is to work-around current limitations related to our privilege/role management in projects. When a user becomes a member of a project, they get the ability to create/manipulate resources such as virtual instances and volumes, which may not be desirable if your goal is to have a user that only has access to Swift or specific Swift containers/buckets. This is not optimal and we are planning to provide options for better permission granularity, but the currently available work-around is to create a separate project for “Swift usage” which has resources quotas set that prevents it’s users from creating/manipulating non-Swift resources.

The Swift project can be used in two ways - to store/manage Swift containers and/or manage Swift-only users.

How to manage permissions via ACLs

If you want a user to be able to create, manage and upload/download objects for any Swift containers created in the Swift project, inviting and assigning the user the role “Object Store” (known as “swiftoperator” in the API/CLI) in “Management” -> “Access Control” -> “Project Users” should be sufficient.

If you instead want a user to be able to read and/or write to a specific Swift containter created in the Swift project or any another project you have, you will need to invite the user to the Swift project, assign them the role “Project Member” and configure Swift container ACLs for the target container(s). We’ll go through an example below.

In order to configure Swift ACLs you will need:

• A user with the “Object Store” (“swiftoperator”) role in the project that contains the container you want to restrict/provide access to
• The Swift CLI or another API client capable of configuring ACLs (this is currently not supported through Horizon)
• An OpenStack RC file (openrc), “clouds.yml” or environment variables set for authenticating towards the API as the user used to configure ACLs
• Name of the container you want to configure ACLs for
• ID of the project in which the container is stored and the ID for the Swift project if they are not the same (listed under “Identity” -> “Projects” in Horizon or openstack project list via the CLI)
• ID of the user you want to restrict/provide access for (accessible through “Identity” -> “Users” in Horizon as that user)

In the following example we’ll use the Swift CLI to configure read/write/list access to a specific container created in the Swift project

# Using variables here to make it easier to follow/adapt to new service users and Swift containers
$ SWIFT_PROJECT_ID="b71cd232c8544cf28a7d7aad797cafe9"
$ SWIFT_CONTAINER_NAME="test-container-1"
$ TARGET_USER_ID="whatever_id_it_has"

# Explicitly specifying project ID here, in-case you use an OpenRC/clouds.yml file downloaded from your other projects
$ OS_PROJECT_ID="${SWIFT_PROJECT_ID}" swift post "${SWIFT_CONTAINER_NAME}" --read-acl ".rlistings,${SWIFT_PROJECT_ID}:${TARGET_USER_ID}" --write-acl "${SWIFT_PROJECT_ID}:${TARGET_USER_ID}"

If you want to provide/restrict access to a container that has been created in another project, the process is similar:

# Specifying the ID for the other project instead
$ OS_PROJECT_ID="<project-id>" swift post "${SWIFT_CONTAINER_NAME}" --read-acl ".rlistings,${SWIFT_PROJECT_ID}:${TARGET_USER_ID}" --write-acl "${SWIFT_PROJECT_ID}:${TARGET_USER_ID}"

Note: Replace <project-id> with the actual Project ID

If you need any clarification, further guidance or have other questions, feel free to reach out to our support.

Known limitations

Currently, cross-project ACLs don’t work if you want to use the S3 compatibility.

Further reading

Swift ACLs

14 - Swift getting started

Overview

Swift is Elastx object storage in OpenStack. It provides high availability using all our availability zones. It is also encrypted at rest as per default. This guide will help you get started with the basics surrounding Swift object storage. We will be using the Swift-cli and OpenStacks Horizon.

Swift CLI

To use swift cli you’ll need either an application credential or openstack rc file.

Prerequisites

• python-swiftclient installed.
• Application credential. See here on how to get started with application crendentials. This is required if your account is using MFA.
• OpenStack rc file. Cannot be used if your account is using MFA. To download the rc file, log into your project, click on your user at the top right and select OpenStack RC File.

Getting started

Start by sourcing your application credential or openstack rc file.

Beware: The following commands executes directly without any questions.

Check your authentication variables

Check what the swift client will use as authentication variables swift auth.

Create your first container

Lets create your first container by using following command:
swift post <container name>.

  $ swift post my_container

Upload files

Upload a file to your container: swift upload <container_name> <file_or_folder> .

  $ swift upload my_container ./file1.txt
  file1.txt

Show containers

To show all your containers, use the following command: swift list.

  $ swift list
  my_container

Show objects inside your container: swift list <container_name>.

  $ swift list my_container
  file1.txt

Show statistics of your containers and objects

You can see statistics, ranging from specific objects to the entire account. Use the following command to se statistics of the specific container we created earlier.
swift stat <container_name>.

  $ swift stat my_container
Account: AUTH_7bf53f20d4a2523a8045c42ae505acx
             Container: my_container
               Objects: 1
                 Bytes: 7
              Read ACL:
             Write ACL:
               Sync To:
              Sync Key:
          Content-Type: application/json; charset=utf-8
           X-Timestamp: 1675242117.33639
         Last-Modified: Wed, 01 Feb 2023 09:15:39 GMT
         Accept-Ranges: bytes
      X-Storage-Policy: hdd3
            X-Trans-Id: tx2f1e73d3b29a4aba99c1b-0063da2e2b
X-Openstack-Request-Id: tx2f1e73d3b29a4aba99c1b-0063da2e2b
            Connection: close

You can also type swift stat <container_name> <filename> to check stats of individual files. If you want to se stats from your whole account, you can type swift stat.

Download objects

You can download single objects by issuing the following command:
swift download <container_name> <your_object> -o /path/to/local/<your_object>.

  $ swift download newcontainer file1.txt -o ./file1.txt
  file1.txt [auth 2.763s, headers 2.907s, total 2.907s, 0.000 MB/s]

It’s possible to test downloading an object/container without actually downloading, for testing purposes swift download <container-name> --no-download.

Download all objects from specific container
swift download <container_name> -D </path/to/folder/>.

Download all objects from your account
swift download --all -D </path/to/folder/>.

Delete objects

Delete specific object by issuing the following command:
swift delete <container_name> <object_name>.

  $ swift delete my_container file1.txt
  file1.txt

And finally delete specific container by typing the following:
swift delete <container_name>.

OpenStack Horizon

With Openstacks Horizon you get a good overview over your object storage. There are some limitations in Swifts functionality when using Horizon, to fully take advantage of Swifts functions we recommend you to use the swift cli.
This guide will show you the basics with using Swift object storage in Horizon.

Create your first container

Navigate to “Project” → “Object Store” → “Containers” Here you will see all the containers in your object storage.
Choose +Container to create a new container:

Choose a name for your new container:

You will se that a new container has been added, which date it was created and that it is empty.

Upload your first file

To upload your first file, press the up arrow next to +Folder:

Select the the file you want to upload:

Download file

To download a file, select your container and press Download next to the object.

View details of an object

You can view details of an object such as Name, Hash, Content Type, Timestamp and Size.
Select the down arrow next to Download for the object you want to inspect and choose View Details:

Further reading

• Swift has an s3 compatible API for applications that don’t support the Swift API. You can read more about how to configure Swift for s3cmd here.
• You can find an S3/Swift REST API comparison matrix here at the OpenStacks documentation.
• If you want to use more advanced features, please see the OpenStacks documentation for Swift.
• Rclone is a good choice if you need more advanced functions while using Swift. You can read more about Rclone’s support for Swift here.

15 - Swift S3 compatibility

Overview

Swift provides an S3 compatible API for applications that don’t support the Swift API. Note that you need to create EC2 credentials for this to work.

NOTE: The S3 region must be set to “us-east-1” for compatibility with “AWS Signature Version 4”

NOTE: If the application does support Swift natively, using Swift will provide superior performance and generally a better experience.

Example s3cmd configuration

The configuration below works with s3cmd:

[default]
access_key = 00000000000000000000000000000
secret_key = 00000000000000000000000000000
host_base = swift.elastx.cloud
host_bucket = swift.elastx.cloud
use_https = True
bucket_location = us-east-1

Known Issues

The Swift S3 API has a known bug with sigv4-streaming and chunked uploads. We are aware of an upstream fix that has been merged - but we do not yet have it implemented on our platform.

16 - Terraform Backend

Overview

Swift is accessable with the s3 backend. To get the access and secret key follow this guide. EC2 credentials

Example configuration

This is what you need to get the s3 backend to work with swift.

backend "s3" {
  bucket = "<The bucket you want to use>"
  key    = "<Path and name to tf state file>"
  endpoint   = "https://swift.elastx.cloud"
  sts_endpoint = "https://swift.elastx.cloud"
  access_key = "<Puth your access key here>"
  secret_key = "<Put your secret key here>"
  region = "us-east-1"
  force_path_style = "true"
  skip_credentials_validation = "true"
}

key variable example: “path/to/tf-state-file”.

This is the path in the bucket.

17 - Volume Attachment Limits

Overview

If you need to attach more than the default limitation of volume attachments, then this is possible by using a custom image and changing the properties of the image to use a non-default scsi driver.

Please note that any servers created before the image properties are set need to be re-created in order to use the updated driver.

The following properties need to be set in order to achieve this.

hw_disk_bus=scsi
hw_scsi_model=virtio-scsi

Volume attachment limits

Openstack CLI examples

Creating a new image (note that additional options are also needed when creating images, please refer to the Openstack documentation for more information)

openstack image create --property hw_disk_bus=scsi --property hw_scsi_model=virtio-scsi ${IMAGE_ID}

Updating an existing private image

openstack image set --property hw_disk_bus=scsi --property hw_scsi_model=virtio-scsi ${IMAGE_ID}

18 - Volume Backup & Restore

Overview

This guide will help you getting started with Volume Backup and Restore in OpenStack’s Horizon and CLI.
Get more information about the OpenStack command-line client.

Create volume backup from Horizon

Navigate to “Project” → “Volumes” pick the volume you want to backup and choose Create Backup.

Choose Backup Name

In the pop-up window, choose a name and a description for your backup and press Create Volume Backup.

Check the status of your backup

Navigate to “Project” → “Volumes” → “Backups” to see the status of the volume.

Restore volume from backup

Backup of a volume can be restored in two ways, one way is to create a new volume manually from the “Project” → “Volumes”, or to have the volume automatically created when restoring from the “Project” → “Volumes” → “Backups”.

Beware: If option two is chosen, the Availability Zone and Size gets chosen automatically. This means that the volume might be added to a different Availability Zone than intended.

Manually Create a new volume and restore a backup to that volume

Navigate to “Project” → “Volumes” and press Create Volume.

Choose a name and description for the new volume.

Beware: Volume Size has to be at minimum the size of the backup. The Volume also has to be in the same Availability Zone as the instance it will be attached too.

Restore

Navigate to “Project” → “Volumes” → “Backups” and press Restore Backup.

Choose the newly created volume and press Restore Backup to Volume.

Automatically create a new volume and restore a backup to that volume

Navigate to “Project” → “Volumes” → “Backups” and press Restore Backup.

Select Create a New Volume and press Restore Backup to Volume.

The restored backup will be available in “Project” → “Volumes”.

Volume Attach & Detach in Horizon

Navigate to “Project” → “Volumes” and press the ⬇ arrow next to Edit Volume on the volume you want to attach and then press Manage Attachments

In the pop-up window choose an instance you want to attach the restored volume to.

Check volumes again in “Project” → “Volumes” to see if the volume is attached to the instance.

To detach the volume, Navigate to “Project” → “Volumes” and press the ⬇ arrow next to Edit Volume on the volume you want to detach and then press Manage Attachments. In the pop-up window choose the instance you want to detach the volume from.

Volume Backup using OpenStack CLI

List all available volumes: openstack volume list.

openstack volume list
+--------------------------------------+--------------------------------------+--------+------+-------------------------------------+
| ID                                   | Name                                 | Status | Size | Attached to                         |
+--------------------------------------+--------------------------------------+--------+------+-------------------------------------+
| 3af38568-20fc-4c36-bca4-72555a6761e4 | 3af38568-20fc-4c36-bca4-72555a6761e4 | in-use |   20 | Attached to MyInstance on /dev/vda  |
+--------------------------------------+--------------------------------------+--------+------+-------------------------------------+

Create backup of your volume: openstack volume backup create <Volume ID or Name> --name <Name of Backup> --description <Description of your Backup>.

Beware: If the volume is attached to an instance, the flag --force has to be added.

openstack volume backup create 3af38568-20fc-4c36-bca4-72555a6761e4 --force --name MyVolumeBackup --description backup_of_my_volume
+-------+--------------------------------------+
| Field | Value                                |
+-------+--------------------------------------+
| id    | 1c06756f-ebe1-4efd-a419-e94184f86fe8 |
| name  | MyVolumeBackup                       |
+-------+--------------------------------------+

When the creation is finished the backup status will show as: available.

openstack volume backup list
+--------------------------------------+----------------+---------------------+-----------+------+-------------------+--------------------------------------+---------------+
| ID                                   | Name           | Description         | Status    | Size | Availability Zone | Volume                               | Container     |
+--------------------------------------+----------------+---------------------+-----------+------+-------------------+--------------------------------------+---------------+
| 1c06756f-ebe1-4efd-a419-e94184f86fe8 | MyVolumeBackup | backup_of_my_volume | available |   20 | sto1              | 3af38568-20fc-4c36-bca4-72555a6761e4 | volumebackups |
+--------------------------------------+----------------+---------------------+-----------+------+-------------------+--------------------------------------+---------------+

Create a new volume to restore to from backup:
openstack volume create <ID or Name> --availability-zone <sto1/2/3> --size <GiB> --description <a description>.

Beware: Volume Size has to be at minimum the size of the backup. The Volume also has to be in the same Availability Zone as the instance it will be attached too.

openstack volume create my_volume_restore --availability-zone sto1 --type 16k-iops --size 20 --description restored_from_backup
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | sto1                                 |
| bootable            | false                                |
| consistencygroup_id | None                                 |
| created_at          | 2022-05-30T07:18:52.000000           |
| description         | restored_from_backup                 |
| encrypted           | False                                |
| id                  | f63a7a2d-7321-49e3-b909-d49fee733f21 |
| multiattach         | False                                |
| name                | my_volume_restore                    |
| properties          |                                      |
| replication_status  | None                                 |
| size                | 20                                   |
| snapshot_id         | None                                 |
| source_volid        | None                                 |
| status              | creating                             |
| type                | 16k-IOPS                             |
| updated_at          | None                                 |
| user_id             | a2d55d905e05459d84ffd96900c25e9d     |
+---------------------+--------------------------------------+

Volume Restore using OpenStack CLI

Restore backup to the newly created volume: openstack volume backup restore <Backup ID or Name> <Volume ID or Name>.

openstack volume backup restore MyVolumeBackup my_volume_restore
+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| backup_id   | 1c06756f-ebe1-4efd-a419-e94184f86fe8 |
| volume_id   | f63a7a2d-7321-49e3-b909-d49fee733f21 |
| volume_name | my_volume_restore                    |
+-------------+--------------------------------------+

After the backup is restored, it will show as: available.

openstack volume list
+--------------------------------------+--------------------------------------+-----------+------+-------------------------------------+
| ID                                   | Name                                 | Status    | Size | Attached to                         |
+--------------------------------------+--------------------------------------+-----------+------+-------------------------------------+
| f63a7a2d-7321-49e3-b909-d49fee733f21 | my_volume_restore                    | available |   20 |                                     |
| 3af38568-20fc-4c36-bca4-72555a6761e4 | 3af38568-20fc-4c36-bca4-72555a6761e4 | in-use    |   20 | Attached to MyInstance on /dev/vda  |
+--------------------------------------+--------------------------------------+-----------+------+-------------------------------------+

Volume Attach & Detach using Openstack CLI

List available volumes: openstack volume list.

openstack volume list
+--------------------------------------+--------------------------------------+-----------+------+-------------------------------------+
| ID                                   | Name                                 | Status    | Size | Attached to                         |
+--------------------------------------+--------------------------------------+-----------+------+-------------------------------------+
| f63a7a2d-7321-49e3-b909-d49fee733f21 | my_volume_restore                    | available |   20 |                                     |
| 3af38568-20fc-4c36-bca4-72555a6761e4 | 3af38568-20fc-4c36-bca4-72555a6761e4 | in-use    |   20 | Attached to MyInstance on /dev/vda  |
+--------------------------------------+--------------------------------------+-----------+------+-------------------------------------+

List available instances: openstack server list.

openstack server list
+--------------------------------------+------------+--------+-----------------------------------------+-------+------------+
| ID                                   | Name       | Status | Networks                                | Image | Flavor     |
+--------------------------------------+------------+--------+-----------------------------------------+-------+------------+
| 864db2db-9c19-416e-aa9d-fc7d713db36c | MyInstance | ACTIVE | test-network=192.168.1.9				   |       | v1-micro-1 |
+--------------------------------------+------------+--------+-----------------------------------------+-------+------------+

Attach your restored volume to an instance: openstack server add volume <Instance ID or Name> <Volume ID or Name>.

openstack server add volume MyInstance my_volume_restore

Check to see if your volume is attached to your instance: openstack volume list.

openstack volume list
+--------------------------------------+--------------------------------------+--------+------+-------------------------------------+
| ID                                   | Name                                 | Status | Size | Attached to                         |
+--------------------------------------+--------------------------------------+--------+------+-------------------------------------+
| f63a7a2d-7321-49e3-b909-d49fee733f21 | my_volume_restore                    | in-use |   20 | Attached to MyInstance on /dev/vdb  |
| 3af38568-20fc-4c36-bca4-72555a6761e4 | 3af38568-20fc-4c36-bca4-72555a6761e4 | in-use |   20 | Attached to MyInstance on /dev/vda  |
+--------------------------------------+--------------------------------------+--------+------+-------------------------------------+

Detach your volume from an instance: server remove volume <Instance ID or Name> <Volume ID or Name>.

openstack server remove volume MyInstance my_volume_restore

Confirm the detachment: openstack volume list.

openstack volume list
+--------------------------------------+--------------------------------------+-----------+------+-------------------------------------+
| ID                                   | Name                                 | Status    | Size | Attached to                         |
+--------------------------------------+--------------------------------------+-----------+------+-------------------------------------+
| f63a7a2d-7321-49e3-b909-d49fee733f21 | my_volume_restore                    | available |   20 |                                     |
| 3af38568-20fc-4c36-bca4-72555a6761e4 | 3af38568-20fc-4c36-bca4-72555a6761e4 | in-use    |   20 | Attached to MyInstance on /dev/vda  |
+--------------------------------------+--------------------------------------+-----------+------+-------------------------------------+

19 - Volume migration

Overview

To migrate volume data between Availability Zones (sto1|sto2|sto3) you can use Openstacks backup functionality. This backup process uses our Swift object storage, which is available across all Availability Zones.

1. Shutdown the instance whose volume will change Availability Zone. Let’s say it’s in sto1 now.
2. Take a backup of the volume (this may take some time, depending on the size of the volume).
3. Create a new volume in Availability Zone sto2 and select the backup as the source.
4. Create a new instance in Availability Zone sto2 and attach the newly created volume.

To get a more in-depth look at how to perform backup and restore of a volume, follow our Volume Backup & Restore guide.

20 - Volume Retype

Overview

This guide will help you getting started with changing volume type in OpenStack’s Horizon and CLI, by using the retype function.

In this example, we will use a detached volume with the type 16k-IOPS-enc.

Get more information about the OpenStack command-line client.

Volume retype from Horizon

Navigate to “Project” → “Volumes” choose the volume you want to retype and press the ⬇ arrow next to Edit Volume and select Change Volume Type.

Choose Type

In the pop-up window, choose a new type and set Migration Policy to On Demand.

Check status

The volume status will change to retyping, this can take a while depending on the volume size. After everything is done, the volume will have the status available.

Volume retype using OpenStack CLI

List all available volumes: openstack volume list --long.

openstack volume list --long
+--------------------------------------+------------------+-----------+------+--------------+----------+--------------------------------------+--------------------------------------+
| ID                                   | Name             | Status    | Size | Type         | Bootable | Attached to                          | Properties                           |
+--------------------------------------+------------------+-----------+------+--------------+----------+--------------------------------------+--------------------------------------+
| ad2ca224-78e0-4930-941e-596bbea05b95 | encrypted-volume | available |    1 | 16k-IOPS-enc | false    |                                      |                                      |
| db329723-1a3e-4fb9-be07-da6e0a5ff0b1 |                  | in-use    |   20 | 4k-IOPS      | true     | Attached to docker-test on /dev/vda  | attached_mode='rw', readonly='False' |
+--------------------------------------+------------------+-----------+------+--------------+----------+--------------------------------------+--------------------------------------+

Retype volume with: openstack volume set --type <volume-type> --retype-policy on-demand <Volume ID or Name>.

openstack volume set --type 16k-IOPS --retype-policy on-demand ad2ca224-78e0-4930-941e-596bbea05b95

openstack volume list --long
+--------------------------------------+------------------+----------+------+--------------+----------+--------------------------------------+--------------------------------------+
| ID                                   | Name             | Status   | Size | Type         | Bootable | Attached to                          | Properties                           |
+--------------------------------------+------------------+----------+------+--------------+----------+--------------------------------------+--------------------------------------+
| ad2ca224-78e0-4930-941e-596bbea05b95 | encrypted-volume | retyping |    1 | 16k-IOPS-enc | false    |                                      |                                      |
| db329723-1a3e-4fb9-be07-da6e0a5ff0b1 |                  | in-use   |   20 | 4k-IOPS      | true     | Attached to docker-test on /dev/vda  | attached_mode='rw', readonly='False' |
+--------------------------------------+------------------+----------+------+--------------+----------+--------------------------------------+--------------------------------------+

When retyping is done, status will be shown as: available.

openstack volume list --long
+--------------------------------------+------------------+-----------+------+----------+----------+--------------------------------------+--------------------------------------+
| ID                                   | Name             | Status    | Size | Type     | Bootable | Attached to                          | Properties                           |
+--------------------------------------+------------------+-----------+------+----------+----------+--------------------------------------+--------------------------------------+
| ad2ca224-78e0-4930-941e-596bbea05b95 | encrypted-volume | available |    1 | 16k-IOPS | false    |                                      |                                      |
| db329723-1a3e-4fb9-be07-da6e0a5ff0b1 |                  | in-use    |   20 | 4k-IOPS  | true     | Attached to docker-test on /dev/vda  | attached_mode='rw', readonly='False' |
+--------------------------------------+------------------+-----------+------+----------+----------+--------------------------------------+--------------------------------------+

21 - Windows volume offline after restart

Issue description

In certain circumstances, extra disks can be marked as offline after a hard reboot is performed and have to be manually brought online again.

By default Windows machines will have the SAN policy set to be “Offline Shared”, thus in most virtualization platforms this issue will occur when a disk is completely shut down and started again.

The reason behind this is that the disk may be assigned to a different virtual PCI device in the underlying host which causes the OS to block the volume from going online automatically when using the default policy.

Change SAN policy

The easiest way to change the SAN policy is using diskpart in a command shell or powershell prompt.

1. Enter diskpart

   diskpart
   

2. To display the current SAN policy, simply run the following command

   san
   

3. Change SAN policy

   san policy=OnlineAll
   

Restore offline volume state

When attempting to initialize a volume you may be seeing the following error

The disk is offline because of policy set by an administrator

1. Start by listing disk information to find out which disk is offline.

   list disk
   

2. Select the offline disk.

   select disk 1
   

3. Clear the read-only attribute for the selected disk.

   attributes disk clear readonly
   

4. To check that the read-only attribute has been cleared, run the following command.

   attributes disk
   

5. Bring the selected disk online again.

   online disk