Guard against eventually consistent services

When enabling services, after the waiter returns, list the enabled
services and ensure the ones we enabled are in there. If not, retry. May
not always resolve #1393, but should help. Unfortunately, the real
answer is probably either:

1. For us to try and get the API updated to only return the waiter when
the service will consistently be available. I don't know how feasible
this is, but I'm willing to open a ticket.
2. For us to build retries into ~all our resources to retry for a set
amount of time when a service not enabled error is returned. This would
greatly slow down the provider in the case of the service legitimately
not being enabled, but is how other providers handle this class of
problem.

Unfortunately, due to the eventual consistency at play, this is a hard
issue to reproduce and prove, though it matches with my
experience--while testing this patch, one of the tests failed with the
error that the serviceusage API hadn't been enabled, but only on step 4
of the test, when calls had already succeeded. Which suggests eventual
consistency, to me. Regardless, this patch shouldn't _hurt_ and should
mostly be an imperceptible change to users, and should make instances
like #1393 less likely.
This commit is contained in:
Paddy Carver 2018-05-23 04:01:05 -07:00
parent cac3502bac
commit 4ec59fccea

View File

@ -6,6 +6,7 @@ import (
"strings"
"github.com/hashicorp/terraform/helper/schema"
"google.golang.org/api/googleapi"
"google.golang.org/api/serviceusage/v1beta1"
)
@ -227,6 +228,30 @@ func enableServices(s []string, pid string, config *Config) error {
if waitErr != nil {
return waitErr
}
services, err := getApiServices(pid, config, map[string]struct{}{})
if err != nil {
return err
}
var missing []string
for _, toEnable := range s {
var found bool
for _, service := range services {
if service == toEnable {
found = true
break
}
}
if !found {
missing = append(missing, toEnable)
}
}
if len(missing) > 0 {
// spoof a googleapi Error so retryTime will try again
return &googleapi.Error{
Code: 503, // haha, get it, service unavailable
Message: fmt.Sprintf("The services %s are still being enabled for project %q. This isn't a real API error, this is just eventual consistency.", strings.Join(missing, ", "), pid),
}
}
return nil
}, 10)
if err != nil {