Should not fail and let director retry on vcloud bosh account limitation

[poblin-orange]

vcloud Iaas provider typically limit the number of simultanous request for a given vcloud user
In that case we get this type of API Error

<Error xmlns="http://www.vmware.com/vcloud/v1.5" minorErrorCode="OPERATION_LIMITS_EXCEEDED" message="[ 17618e03-603e-4ff0-a323-4a553e8ea452 ] The maximum number of simultaneous operations for user &quot;bosh&quot; on organization &quot;Cube4PaasQA&quot; has been reached." majorErrorCode="400" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsichemaLocation="http://www.vmware.com/vcloud/v1.5 http://normandie-cube-api.itn.ftgroup/api/v1.5/schema/master.xsd"></Error>    

As the error is not permanent, CPI should retry, or let Bosh Director know that he can retry the vm creation.

[cppforlife]

@poblin-orange currently we set https://bosh.io/jobs/director?source=github.com/cloudfoundry/bosh&version=207#p=director.max_threads to 4 for vcloud to avoid overwhelming vcloud. do you already do that?

[poblin-orange]

@cppforlife yes we limit on director side. However, we used a single bosh account for our deployment (1 microbosh => 1 master bosh => 2 cloudfoundry/ service bosh ). In that case, vcloud requests can exceed the 4 per bosh director configuration.

[cppforlife]

Ah makes sense. Also github hid the error returned -- fixed it.

[cppforlife]

Related material found on google: http://infrastructureadventures.com/2012/08/11/vcloud-director-policies-part-2-quotas-limits/

[jmhardison]

Raising this one again.
In an on-premises installation of vCloud Director, there can be a low user quota assigned. In these cases, failing on user quota being exceeded results in a overall failure both in deployment and cleanup. In cleanup, several vApps are left some empty, and some with powered off guests.

As an example, our user quota is currently 2. Any user attempting to submit more operations than allowed will receive a temporary error until other queued jobs finish.

Defining the director threads at 1 does not alleviate the issue as it appears the CPI is firing many more request at once than the defined thread count.