Scenario
Deploying an application with Stateful services, results in Stateful Service stuck in pending state for long. Examining the Service replica log shows the error :
Failed to provision volume with StorageClass "gp-landingpage-sc-uj2z": InvalidZone.NotFound: The zone 'us-east-1a' does not exist.
pod has unbound immediate PersistentVolumeClaims
0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
no nodes available to schedule pods
This happens when the Service is scheduled on a nodepool in a different availability zone where as the Volume Provision Policy provisions the volume in a different availability zone.
Resolution
Modify the Volume Provision Policy to the availability zone to match the node pool availability zone.