Every team we talk to has the same monitoring story. Someone set up checks in a dashboard two years ago. The service has been rewritten since. Nobody is completely sure which of the fourteen alerts still mean anything, and nobody wants to be the person who deletes the one that mattered.
Put the checks in the repository
Defining checks as Terraform resources next to the service they watch fixes this by making the check part of the change. Rename an endpoint and the check that hits it is in the same diff. Delete a service and its checks go with it.
A minimal module
Start with three things per service: an API check on the health endpoint, a browser check on the single most important user flow, and an alert channel. Resist the urge to add more until something has actually broken without you noticing.
Run them from more than one region
A check that only runs from one region tells you about that region. Two regions catches most routing and certificate problems. Three is usually noise unless you actually serve three regions.
Review them like code
The point of this is not that Terraform is nicer than a dashboard. It is that a pull request has a reviewer, and a dashboard does not.
