The combination SSO and vCAC 6.0, regardless whether the vCAC Identity Appliance is used, a VMware SSO appliance (as part of the vCenter appliance) or a vCenter SSO installation on Windows, results in a 90-day “time bomb”. The SSO internal administrator account for a tenant has a password that expires after 90 days. This has been described, with a solution in knowledge base article KB 2075011 “vCloud Automation Center 6.0.x tenants become inaccessible and identity stores disappear”. To my surprise however, when this happened yesterday (exactly 90 days after an upgrade of vCAC), the solution in the KB did not work. What we did see was that a new user ID appears in the vCenter Server when looking with the webclient at Administration, Single sign On, Users and groups, DCAdmins, that a new user appeared there called tenantAdmin.
That in itself is as meant to be. The new user ID is created by running the LDIFDE commands with the UserAccountControl.ldif and PasswordExpiration.ldif files you create when following the KB for a windows based installation of vCenter SSO. Only, it doesn’t solve the issue.
The logfile vmware-sts-idm.log on the vCenter SSO server gave us more information:
2014-08-28 15:05:05,551 WARN [ServerUtils] cannot bind connection: [ldap://localhost:11711, CN=Administrator,CN=Users,DC=xxxxx]
2014-08-28 15:05:05,551 ERROR [ServerUtils] cannot establish connection with uri: [ldap://localhost:11711]
2014-08-28 15:05:05,551 ERROR [IdentityManager] Failed to find solution user by subject DN [CN=cafe-43aa459e-88ae-4363-a55b-8f67508f47cc] in tenant [xxxxx]
2014-08-28 15:05:05,552 ERROR [ServerUtils] Exception ‘com.vmware.identity.interop.ldap.InsufficientRightsLdapException: Insufficient Rights
com.vmware.identity.interop.ldap.InsufficientRightsLdapException: Insufficient Rights
Hmmm, it says Administrator here and not tenantAdmin. So we changed the common name accordingly and ran the LDIFDE commands again for both .ldif files.
where [tenant_name] is the part in the portal url that comes after /org/
We tried logging in to the tenant portal and suddenly we were back in business! I logged off and logged in as system administrator on the default tenant and checked whether the identity store for the tenant was there, and Yes!, it had reappeared. Just to be sure I ran a couple of tests to check whether everything was working again, which it fortunately did.
So, just to recap, the problem was that we were using the wrong CN in both the passwordExpiration file and the UserAccountControl file. We were using cn=tenantAdmin, as per the KB, where as SSO logs show the CN is actually Administrator. Changing both files to reflect this resolved the issue. This has been passed to the people responsible for the knowledge base and this will be updated.
My advice, don’t wait for the 90 days to pass, your tenants won’t like this happening. After you have set up a new tenant, run the LDIFDE commands with the change described above to disable expiration of the password. You have to do this for each and every tenant you create. So why not build this into your “Create a tenant automatically”-workflow?
And big thanks to Gary Doyle for his support with this case!
One of our customers complained of a disconnect – reconnect every 30 seconds of sessions with virtualized desktops (Horizon View 5.3.1). We tried it ourselves and got the same results. After about 10 disconnect – reconnects we got the message in the Blast portal “unable to reconnect”. With a security specialist and a network specialist we started to troubleshoot, aiming to isolate the problem. We have a setup with a virtual wire for the VDI desktops and another vWire for the servers they have access to, we use the Blast portal to publish the virtual Windows 7 desktops for the customer on a private network. Access is fenced of with a firewall on the outside perimeter, a TMG to publish the URL of the Blast portal, and for the translation of the Blast portal port to 443 we use a Sophos appliance. Not really a very simple setup.
Anyway as the Blast portal didn’t suffer from the time outs and disconnects, we could rule out the TMG pretty fast. Soem other simple test showed that the problem couldn’t be in the Sophos appliance either. Then we continued our tests with rdp sessions between the virtual desktops. There we found no problems. Then RDPto machines on different virtual wires… and we got freezing sessions. So we could rule out Horizon View Broker and Agent as the cause of all evil.
We set off to examine the vCNS Edges… and there we found a problem. The autogenerated default internal rules were missing from the responsible Egde, so we had no HA rules present on the Edge, causing connection problems.
The solution was to enable auto rule generation…
And voilà, problem solved!
way too late. But I had some serious problems at hand, which I’ll write about in a separate blogpost. Anyway, de 2nd keynote elaborated on the first. More details were released about future features in vSphere 6 (for those taking part in the Beta* no surprise) like 4 vCPU FT, VVOLs, much improved long distance vMotion, vMotion across vCenter Servers (Ooh, I particularly like that one) and more details about the new VMware AIR and EVO programs. In the mean time we’ll have to do with vCloud Suite 5.8, SRM 5.8, and VDPA 5.8.
vCloud Suite 5.8
Although this update is not as radical as we would have liked, the new features could be resumed as follows:
- Improvements in business continuity (BC or BCP – the P denotes planning) and disaster recovery (DR) by way of self-service, policy-based provisioning of DR tiers and increased scalability of protection and recovery capabilities.
- Big Data Extensions supports Hadoop 2
- Interoperability with NSX with customizable provisioning of NSX firewall and routing services
- Proactive support provided by Support Assistant, a free vCenter plug-in that attempts to identify issues before problems occur.
VMware Site Recovery Manager 5.8
VMware vCentre Site Recovery Manager (SRM) 5.8 aims to deliver next-generation disaster recovery capabilities for vSphere environments.
The main highlights of this new release are:
- 5x the scale of protection – IT organizations can set up recovery plans scalable up to 5,000 virtual machines per vCenter Server using array-based replication to enable enterprise-level protection–five times larger than with previous limits.
- Enhanced self-service – New integrations will offer customers self-service access to provision predefined disaster recovery protection tiers to new VMs via blueprints in vCloud Automation Center when using array-based replication.
vSphere Data Protection Advanced 5.8
vSphere Data Protection Advanced 5.8, powered by EMC Avamar technology, offers backup capabilities for vSphere environments.
The new major features in vSphere DPA are:
- Enhanced support for business-critical apps – customers can now perform backups for Microsoft SQL Server clusters as well as Microsoft Exchange DAGs (Database Availability Groups).
- Customizable Proxies and Backup Work Streams – customers now have the flexibility of choosing the number of parallel backup workstreams to satisfy Service Level Agreements (SLAs) helping them to increase backup scalability and performance.
- Enhanced Replication – customers can now restore replicated backups at either primary or disaster recovery site.
* In Old Greek, the alphabeth’s second letter was pronounced “bettah” (phonetic alphabet: [b’eta]), not with a diphthong in the first syllable as in “baytah” (phonetic alphabet: [b’ejta]) and not with an ee sound (phonetic transcription [b’i:ta (thanks to Jessica Goethe for the info).
So, even though I cannot be physically present at VMworld this year as I had to cancel flight, hotel and so on as mentioned in my previous post, I’ll attempt at least to cover the keynote speeches. Already there have been a lot of post about rebranding of suites and products, like the new vRealize suite.
Anyway, NO LIMITS!
Opening speaker was Robin Matlock, Chief Marketing Officer. After some attention to the earthquake this weekend, she mentioned over 22.000 attendees (minus me) this year! And she talkes about change… “Change is either a barrier or an opportunity” leading to building the Golden Gate Bridge.
Next up: Pat Gelsinger, VMware’s CEO
He talked about VMware’s Brave Journey and announced new developments:
- VMware vCloud Suite 5.8,
- vSphere 6.0 Beta, Virtual Volumes (Vvol) and vSAN 2.0 Beta
- Rebranding and bundeling the operations and automation suites to the vRealize suite
- VMware EVO, a family of hyper converged infrastructure solutions, delivered through partners. Its first member is EVO:Rail, designed for up to 100 VM’s. A solution similar to Nutanix, but with partners like EMC, Dell. The next family member will be EVO:Rack (Tech Preview), bringing the power of EVO at a data center scale, cloud scale. More info can be found at http://vmw.re/1veRQoD
- VMware joins OPC (Open Compute Project)
- A VMware Open Stack integration (now in Beta)
- partnerships with Google and Docker, leading to one platform for any app
- the Workspace Suite, combining Horizon with Airwatch and Content Locker in one workspace portal
- rebranding of vCloud Hybrid Services to vCloud Air
After that he introduced Bill Fathers, EVP an GM of Cloud Services.
Cloud computing makes companies more agile, more efficient and realize huge cost savings and benefits. In 2009 25 of VM’s lived in the cloud. In 2014 6% of all VM’s live in the cloud. In the same time span the amounts of VM’s increased hugely. The increase of 2% to only 6% is therefore misleading.
Bill announced several new services:
- DevOps Services – Continuous-Integration-as-a-Service
- Database-as-a-Service – starting with MySQL and MS SQL Server
- Object Storage – Beta will be launched in September
- Mobility Services
- and Cloud Management – vRealize Air Automation – Beta in Q4
- A new Beta program, Ondemand; see vmware.com/go/ondemand
Final keynote speaker of day 1 was Carl Eschenbach, COO and President of VMware
Carl spoke about the brave new world of IT and then showed three customer cases: Medtronic, MIT and Ford. Finally Carl introduced VMware EVO RAIL with some early stage customers stating their opinion about EVO and EVO RAIL.
It is about the power of “AND”, VMware & You. Endless Possibilities.
Unfortunately I had to cancel my VMworld 2014 participation and all appointments I made 10 days before VMworld 2014 starts. The current project I am working on has reached a critical stage and I cannot be missed on site. Bummer. Anyway, here is a list of the sessions I chose and I would like to use this blog to ask you all a question I had reserved for the VMware consultants at the expo.
VMworld 2014 US sessions:
5K fun run/walk
PAR2987 Software-Defined Data Center Executive Session: Vision and Strategy
TEX2254 VMware NSX Extensibility: Enabling Partner Integrations
TEX2730 What’s New in vCloud Suite?
STO2197 Storage DRS: Deep Dive and Best Practices
MGT1969 vCloud Automation Center and NSX Integration Technical Deep Dive
MGT2525 Chasing the White Rabbit all the Way to Wonderland: Extending vCloud Automation Center Requests with vCenter Orchestrator
PAR2218 Partner EUC 202: Mobility (AirWatch) Strategy and Roadmap
SDDC1176 Ask the Expert vBloggers
NET1589 Reference Design for SDDC with NSX & vSphere
SDDC1600 Art of IT Infrastructure Design: The Way of the VCDX – Panel
NET1974 Multi-Site Data Center Solutions with VMware NSX
NET1674 Advanced Topics & Future Directions in Network Virtualization with NSX
PAR2982 Partner SDDC 205: Driving Change with NSX
NET1861 Automating Networking and Security Services with NSX for vSphere and vCenter Orchestrator (vCO)
NET2379 Dynamically Configuring Application Specific Network Services with vCAC and NSX
TEX2044 VMware Compliance Reference Architecture Framework Overview for Partners
INF2427 DRS : Advanced Concepts, Best Practices and Future Directions
BCO2701 vSphere HA Best Practices and FT Tech Preview
MGT2175 vCloud Automation Center Overview and Glimpse into the Future
As you can see, most sessions I wanted to take are around vCAC and NSX, topics very real for my current assignment so I will miss the info dearly. Of course There were some some parties in my schedule too, ideal places to do some innocent networking, like VMunderground, the PernixData party, the Tegile party, the Nutanix party, the Veeam party, the vExpert/VCDX party and the VMworld party with the Black Keys which I all have to miss. Fortunately one of my colleagues and fellow vExpert will pick up my vExpert gifts, I hope he’ll reserve some space in his suitcase for me.
The big question:
So, now to the question I wanted to raise at VMworld, and I hope you can shed some light on this: When migrating a vCloud Automation Center 6.x environment from a single tier setup to a distributed setup, can I move the PostgreSQL database to the new PostgreSQL database cluster without losing my existing tenant? The idea is that we move it after building the distributed setup (hopefully vCAC 6.1 will be available by that time), but before we create new tenants and/or workflows. Please use the form below, I hope you can help me out, many thanks in advance.