vCAC 6.0 Tenant Identity store disappearing (vRealize Automation)
The combination SSO and vCAC 6.0, regardless whether the vCAC Identity Appliance is used, a VMware SSO appliance (as part of the vCenter appliance) or a vCenter SSO installation on Windows, results in a 90-day “time bomb”. The SSO internal administrator account for a tenant has a password that expires after 90 days. This has been described, with a solution in knowledge base article KB 2075011 “vCloud Automation Center 6.0.x tenants become inaccessible and identity stores disappear”. To my surprise however, when this happened yesterday (exactly 90 days after an upgrade of vCAC), the solution in the KB did not work. What we did see was that a new user ID appears in the vCenter Server when looking with the webclient at Administration, Single sign On, Users and groups, DCAdmins, that a new user appeared there called tenantAdmin.
That in itself is as meant to be. The new user ID is created by running the LDIFDE commands with the UserAccountControl.ldif and PasswordExpiration.ldif files you create when following the KB for a windows based installation of vCenter SSO. Only, it doesn’t solve the issue.
The logfile vmware-sts-idm.log on the vCenter SSO server gave us more information:
2014-08-28 15:05:05,551 WARN [ServerUtils] cannot bind connection: [ldap://localhost:11711, CN=Administrator,CN=Users,DC=xxxxx]
2014-08-28 15:05:05,551 ERROR [ServerUtils] cannot establish connection with uri: [ldap://localhost:11711]
2014-08-28 15:05:05,551 ERROR [IdentityManager] Failed to find solution user by subject DN [CN=cafe-43aa459e-88ae-4363-a55b-8f67508f47cc] in tenant [xxxxx]
2014-08-28 15:05:05,552 ERROR [ServerUtils] Exception ‘com.vmware.identity.interop.ldap.InsufficientRightsLdapException: Insufficient Rights
com.vmware.identity.interop.ldap.InsufficientRightsLdapException: Insufficient Rights
Hmmm, it says Administrator here and not tenantAdmin. So we changed the common name accordingly and ran the LDIFDE commands again for both .ldif files.
where [tenant_name] is the part in the portal url that comes after /org/
We tried logging in to the tenant portal and suddenly we were back in business! I logged off and logged in as system administrator on the default tenant and checked whether the identity store for the tenant was there, and Yes!, it had reappeared. Just to be sure I ran a couple of tests to check whether everything was working again, which it fortunately did.
So, just to recap, the problem was that we were using the wrong CN in both the passwordExpiration file and the UserAccountControl file. We were using cn=tenantAdmin, as per the KB, where as SSO logs show the CN is actually Administrator. Changing both files to reflect this resolved the issue. This has been passed to the people responsible for the knowledge base and this will be updated.
My advice, don’t wait for the 90 days to pass, your tenants won’t like this happening. After you have set up a new tenant, run the LDIFDE commands with the change described above to disable expiration of the password. You have to do this for each and every tenant you create. So why not build this into your “Create a tenant automatically”-workflow?
And big thanks to Gary Doyle for his support with this case!