Tag: VCSA

Using WinSCP with the VMware vCenter Server Appliance

From time to time, you will find it necessary to transfer files to or from your VMware vCenter Server Appliance (VCSA) or ESXi Servers. If you are working from a Windows desktop or server, there’s no more convenient utility than WinSCP for copying files securely between Windows and Linux Systems.

There are a few issues, however, when it comes to connecting to the VCSA with WinSCP that I will show you how to work around without reconfiguring the default shell of your VCSA! Continue reading

VCSA and ESXi password security

vcsa password security

I recently went looking for information on password security for the VCSA 6.0 & 6.5 and ESXi 6.0 & 6.5. Most specifically, I was interest in the number of passwords remembered, so I could define that in documentation for a client.

Try as I might, I couldn’t find documentation for VCSA number of passwords remembered or how to configure it anywhere! Continue reading

VCSA disks become full over time

I’ve recently spoken with a number of VMware vCenter Server Appliance 6 (VCSA) users that have had issues with the root filesystem of VCSA running out of space.

This situation seems to be occurring more often now due to a combination of when the VCSA 6 went mainstream (18 to 24 months ago) and the default 365 day password expiration. The combination is just long enough for the root password to expire and after about 6 months (depending entirely on the size and activity of the vSphere environment) the /dev/sda3 disk fills! Continue reading

Upgrading to VCSA 6 fails

I began an upgrade of the VMware vCenter Server Appliance from 5.5 to 6 for a small (in VMware’s own terminology ‘Tiny’) vSphere environment of 3 hosts and about 30 VMs. I certainly didn’t anticipate any trouble beyond the usual hassles associated with upgrading an infrastructure-level service like vCenter.

Unfortunately, carefully following VMware documented procedures and the best advice some of my favorite blogs had to offer (see: VMware KB 2109772, http://www.vladan.fr/how-to-upgrade-from-vcsa-5-5-to-6-0/, and http://www.virtuallyghetto.com/2015/09/how-to-upgrade-from-vcsa-5-x-6-x-to-vcsa-6-0-update-1.html ), and the upgrade still failed.

When the installer failed, the only message the web-installer had to offer was the well-known: Firstboot script execution error.

Firstboot script execution error

Furthermore, the log files failed to download, leaving me with fewer diagnostic resources than I might otherwise find on my Windows desktop.

Normally, the Firstboot script execution error is the result of incorrectly configured DNS, so the first thing I tested was forward and reverse DNS; both were working perfectly.

I decided to delve further into the issue, and found that the SSH daemon on the VM had started, so I connected with Putty for a look around.

Side-note: upgrading from VCSA 5.5 to VCSA 6 requires the creation of a brand-new VM, then the supposed automatic migration of data, leaving you with your original VCSA in a powered-off while a new VM is intended to take its place.

As long as I was in with SSH (Putty), I did some more poking around, and finally ran: df –h on the VM that was supposed to be my new ‘upgraded’, VCSA 6.

storage_seat_full

The problem was immediately apparent; the /storage/seat partition (virtual disk) was completely full! In VCSA 6, /storage/seat is used for the Postgres Stats Events And Tasks (SEAT).

Side note: VCSA 6 puts all of its primary partitions on separate virtual disks, 11 in total. This is a great advantage for long-term scalability, but somewhat of a disadvantage as compared to one disk where every partition can grow to the capacity of the disk. To learn more about what all of the different partitions/disks do, look at this excellent write-up on virtuallyGhetto: Multiple VMDKs in VCSA 6.0?

What I (and VMware) had failed to take into account in sizing of the VCSA, was the potential for an extraordinary number of Tasks and Events. While this may be a ‘Tiny’ deployment by VMware’s standards, with a Horizon View environment plus Veeam Backup and Replication running on a sub 1-hour R.P.O., the number of Tasks and Events presents more like what VMware seems to expect from a ‘Medium’ deployment.

One potential solution may be to Reclaim or purge data from the Postgres database on the VCSA 5.5 before trying the upgrade; but the owner decided in favor of preserving all of the data if possible.

The Solution

In the end, the solution was simply to select a larger deployment size while going through the web-installer wizard. As it turned out, the /storage/seat disk for a ‘Tiny’ deployment was only 10GB, while it was 50GB for a ‘Medium” deployment.

During the upgrade (which took over 2 hours), I connected via SSH as soon as the daemon had started and ran df –h a number of times (I should’ve used: watch). I saw the /storage/seat volume grow slowly, eventually reaching over 17GB of used space, before settling back to 16GB on the successful upgrade.

The only drawback I can think of, to having specified a ‘Medium’ size deployment for a relatively small environment is that the vCPU and RAM allocated to the VM are now vastly beyond what is required with 24GB RAM and 8 vCPU. I plan to shut the VCSA down and scale back to around 16GB RAM and maybe 4 vCPU, better to suit the environment at my earliest opportunity.

Backing up and restoring the vCenter Server Appliance 6 database

One extremely important advantage of the VMware vCenter Server Appliance (VCSA) is its native PostgreSQL (vPostgres) database. With the embedded database and VCSA, it is now possible to support installations which scale to the maximum capability of vCenter, without additional Operating System or Database licensing costs.

Incumbent with the use of VCSA, however, comes a certain degree of responsibility for backing up the PostgreSQL database embedded with the vCenter Server Appliance. A good backup of the VCSA database makes the following tasks much easier:

  • In-place restore of VCSA database
  • Migration to a different installation of VCSA
  • Protection of vCenter tasks & events for auditing purposes

The process is actually very simple and detailed in VMware KB: 2091961, however typical to VMware, there are few actual procedural details which might help an admin who was not intimately familiar with Linux procedures, for example:

  • How do you create the folder you want to keep database backups on the VCSA?
  • How do you transfer the vCenter vPostgres backup and restore package linux_backup_restore.zip to the VCSA?
  • How do you extract a ZIP archive on a Linux system?
  • How do you retrieve the database backup from the VCSA once it is created?

I will answer these questions in a simple, step-by-step procedure with screenshots and suggestions for applications and settings to use in the process.

Preparing to back up the vCenter Server Appliance (6.X) Database

Open the VMware KB: 2091961 and scroll down to locate the attachment: linux_backup_restore.zip

Save the file to an appropriate folder on your local Windows system

Install WinSCP

Now, if you don’t already have it installed, go get the WinSCP Installation Package. WinSCP is free and one of the most useful utilities with vSphere in general, but there are some WinSCP settings specific to vSphere and the VCSA.

Get the Installation Package so you can save settings specific to your environment

Run as administrator

image005

Choose your language and then: OK

Next

Accept and then Install

Choose to donate and/or Finish

Enable SSH on the VMware vCenter Server Appliance 6

Using a vSphere Client, open a Virtual Machine Remote Console window to your VCSA installation to make sure SSH is enabled.

image010

 

This works just like it does in ESXi:, press F2 to log-in

The password you specify here will be the OS password you set when installing the VCSA.

Using the up/down arrow on your keyboard, scroll down to: Troubleshooting Mode Options then press: Enter

Now, using the up/down arrow, highlight: Enable SSH and press: Enter to toggle SSH for your VCSA installation. NOTE: it is not necessary at this time to enable the BASH Shell, we will do that from Putty

When set correctly, this is how it looks:

Press: Esc to exit

Log in to the vCenter Server Appliance Linux console as root

Open Putty (I am going to take for granted that you already have this one!) and type in the IP address of your VCSA installation (in my example above, it is: 192.168.153.110)

Yes to accept

Enter the username: root and the password that you set for your OS installation.

Now, type (or copy & paste) the two commands to Enable BASH Access and Launch BASH Shell on your VCSA

You will find yourself at the root of the VCSA installation

Now, create a folder to store the PostgresSQL backups, at least until you are able to transfer them off the system. Run the command: mkdir db_backups

Now, list that folder with permissions by typing: ls –la to list the root folder

We can now see the permissions for: db_backups as: drwx (or just rwx for the User)

The Linux permissions listing works as follows:

Read Write eXecute Read Write eXecute Read Write eXecute
d r w x

so the fact that our directory “db_backups” shows “drwx” means that the user (that’s us) has Read, Write and eXecute on this folder.

Change directory into the: db_backups folder

Connecting to VCSA with WinSCP

Using WinSCP successfully with the VMware vCenter Server Appliance requires one of two things to occur:

1.Reconfigure VCSA

–OR-

2.Reconfigure the connection on WinSCP

I universally choose to re-configure WinSCP to work with my vCenter Appliance!

Enter your basic connection parameters in WinSCP and click: Advanced

Now choose: SFTP and enter the following value as SFTP Server: shell /usr/lib64/ssh/sftp-server

Now click: OK and then: Login

Yes

Continue

And you are in

Now locate the location you saved: linux_backup_restore.zip (on the left) and the folder you create on VCSA (on the right)

and drag-and-drop the file to copy to your VCSA

Extract the ZIP on the VCSA

List the contents of the directory: db_backups with the command: ls –la

Unzip: linux_backup_restore.zip

List the contents of the directory: db_backups with the command: ls –la

Neither of the scripsthas eXecute permissions, so add eXecute for the User with the command: chmod +x *.py

List the contents of the directory: db_backups with the command: ls –la (again)

The *.py have become eXecutable!

Backup the VMware vCenter Server Appliance PostgreSQL (vPostgres) database

Run the: backup_lin.py script, provide a filename: python backup_lin.py –f 11112015_VCDB.bak

Now use WinSCP to transfer the backup to a different location

You will have to refresh the folder listing in WinSCP (Ctrl+R) to see the files created

Drag the database backup to your chosen folder on Windows

Restore the VMware vCenter Server Appliance PostgreSQL (vPostgres) database

First, use WinSCP to upload the appropriate backup file. WinSCP will prompt you to overwrite, if a copy of that file exists. Make your choice.

Stop the vCenter Server with: service vmware-vpxd stop

Stop the vCenter Datacenter Content Library Service with: service vmware-vdcs stop

Restore the vCenter database with: python restore_lin.py –f 11112015_VCDB.bak (or whatever the name of your file)

There may be numerous “NOTICE” lines referencing parts of the vCenter Server Appliance which simply don’t exist in your configuration. Just ignore these and look for the ultimate message: Restore completed successfully

Now start the vCenter with: service vmware-vpxd start

Now start the vCenter Datacenter Content Library Service with: service vmware-vdcs start

And you should be in business!

VMware vSphere 6 is finally ready!

Unlike in previous releases, VMware took a good long time getting vSphere 6 ready. For the first time ever, VMware made the Beta version of vSphere 6 publicly available (all you had to do was sign-up) and was actively soliciting input on the Beta forums.

The initial public release of vSphere 6 (3/12/2015) was, nonetheless, plagued with at least one critical issue and several annoyances. The most critical issue affecting vSphere 6 was that Changed Block Tracking (CBT) appeared to be essentially broken, rendering most forms of backup using the vSphere API for Data Protection useless[1]. This and other less significant issues rendered the initial release of vSphere 6 unsuitable for use in production environments, and I recommended that all users hold off on upgrading until these issues had been resolved. Continue reading