(Originally posted at https://docs.google.com/document/d/1kn-M-bFy2RsWoxF5fRESi-hiE8G-5zXe-2I88tJoHhc )

Roue53 DNS Challenge Presentation - rethink

Problem:

Repeated creation of a new ChangeRecord on route53 API when the ChangeRecord does not transition from PENDING to IN_SYNC within 120 seconds. As per AWS documentation this can sometimes even take ~30 mins. As a result, this compounds the situation when Route53 endpoint is going through excessive traffic contributing to additional requests triggering rate limiting.

Manifestations

• https://redhat.atlassian.net/browse/OCPBUGS-42199
• https://redhat.atlassian.net/browse/CM-1205

Earlier work

• As a first reaction, a solution to infinitely wait for ChangeRecord Request status to go INSYC was thought of based on how its implemented for the GCP provider. But we did a bit more analysis as below
• (not same, but related: Correction in initial backoff period: feat(trigger): adding certificate request backoff duration to trigger controller #8312)

Background

Challenge Controller breaks down the provisioning of challenge into 3 steps and delegates it to the external DNS provider:

1. Present.
   • This phase is supposed to call the configured DNS provider to initiate the needed provisioning of the change record..
   • If any failures are seen, the controller retries to "Present" with exponential backoff as a failure here is considered as an API failure etc, giving the time for the backing service to recover..
2. Check.
   • Here, the controller checks if the requested change is propagated (by doing a DNS query to the configured authoritative nameservers and verify the expected TXT record)
   • The authoritative nameservers are fetched from.
     • Either the default recursive nameservers
     • Or the user configured recursive nameservers
     • Example (hypothetical) DNS query steps in this case:-
       • Table A
   • If -dns01-recursive-nameservers-only (default False) is set, then just recursive nameservers are used to query the TXT record
     • Example (hypothetical) DNS query steps in this case:-
       • Table B
   • If any failures are seen here, the controller re-queues with a fixed interval (10s).
3. Accept .
   • Respond to the ACME server to go ahead and do the DNS challenge..

Table A: Example DNS query steps when -dns01-recursive-nameservers-only is False (default)

Table B: Example DNS query steps when -dns01-recursive-nameservers-only is True

If we go by this broader division of work, a failure in "Present" SHOULD be API failures, like any content level status codes that explicitly specify a failure, or an API failure code like 4xx/5xx/6xx response, or a timeout / protocol error etc.

However, the way its coded now, "Present" seems to concentrate all the "DNS Provider API" calls, and "Check" the actual DNS queries.

Because of which, "Present" is kind of forced to consider ChangeRecord Status not moving from Pending to IN_SYNC as a configuration error.

Solutions

Alternative 1

We should respect the "DNS Provider API" response, and if its content status or protocol status of the Create Change Record does not indicate a failure, we should consider that the ChangeRecord is configured.

And, the check for propagation should move to the "Check" phase.

This will need the ChangeId to be remembered by the challenge controller and would not be a trivial change.

If we move the status monitoring to "Check" phase, we won't be creating new ChangeRecords.

And a ChangeRecord "stuck" in Pending state will be an anomaly instead of something that the controller works around.

Alternative 2

The Present phase relies on API response, and does NOT wait for the request to move from Pending to INSYNC. Because, after the Pending->INSYNC wait, in the Check phase, we also perform DNS queries on configured authoritative nameservers to ensure we get back the expected TXT record.. See Check phase in section: Background above.

Ambiguity in alt-2

• In case we don't wait for the INSYC status:.
  • Is there a possibility that the DNS query succeeds from the cluster, but may fail from the ACME Server? (due to network segmentation etc).

Decisions

<TODO>