What you describe at Mark specific CI builds flaky | github-feedback should already be possible using continue-on-error: Workflow syntax for GitHub Actions - GitHub Docs
Prevents a workflow run from failing when a job fails. Set to true to allow a workflow run to pass when this job fails.