Deploy: fixes to support version history properly

[josephjclark]

Several fixes to deploy to take version history into account

• Ensure the version hash algorithm is the same in CLI and Lightning
• Ensure the new workflow_history from app state is tracked
• When checking out a project, write a forked_from map from each workflow into openfn.yaml
• When checking for divergence, ensure forked_from is the head of the workflow in the app (but only for workflows we've changed locally...
• ... and when merging, we only merge the workflows that have changed (just to be sure we don't revert anything)
• Fetch is now destructive and will override your local project file without warning

Note that to work properly, this needs an update on the lighting side. I think I'm gonna sync releases.

Fixes #1231

So in short:

• Deploy, fetch and checkout (? check this!) will all warn you if you're in danger of losing work

TODO:
Need to run a full checkout after the deploy step (so that new workflows and stuff are properly represented)

Testing notes:

• If there is no local change, nothing will be deployed
• Can push a local change to the app
• Can pull a remote change from the app
• If changes are made to the same workflow in the app and CLI, a divergence warning is raised
• A local change on Workflow A is compatible with a remote change on Workflow B

AI Usage

Please disclose how you've used AI in this work (it's cool, we just want to know!):

• Code generation (copilot but not intellisense)
• Learning or fact checking
• Strategy / design
• Optimisation / refactoring
• Translation / spellchecking / doc gen
• Other
• I have not used AI

You can read more details in our Responsible AI Policy

[josephjclark]

So the deploy stuff seems to be working. Got a problem now where I'm getting conflicts every time I pull, even when tehre aren't conflicts (remote is ahead of local so we can just overwrite). This might be because I'm doing a generic merge - but like on deploy, the test is a little different to general merge test

Also I've messed the tests up but that's kind of anticipated - will fix soon

[josephjclark]

Bit of a revelation here after lunch.

if the provisioner is sending correct version histories, and if the version algorithm is the same in CLI and app - then we don't need forked from. The forked_from version should be at the head of each workflow history.

So if you check out a workflow, and generate the version hash for it, you know if the workflow has changed since the last fetch because the local version is not in the workflow history. You also know if the remote version is ahead of unchanged local because the local version is in the remote history, just not at the head.

If you check out locally and made an edit, the local version hash should be different to the head of history.

So I think we have all the information we need to make good decisions here, and shouldn't need to use the fork_version metadata.

Ah no that's not quite right:

• I checkout a branch
• I make a new change
• I now have a divergent local head and a history
• I fetch from the app
• How do I know if the app's history has diverged from my local history? Forked form gives me an anchor - I can see in the remote history where we were when we edited. If we don't have forked from, we know that local head and remote head are different, but we don't know the remote head changed since we checked out.

Phew. Complex. We do still need forked_version.

[josephjclark]

I've been thinking a lot about merge logic over the last couple of days and I'm worried I'm about to start building a lot into the CLI

I think really the Project package needs to support the different kinds of merge: like syncing a local to a remote is a pariticular strategy (we detect locally changed ones, merge those into the remote, and throw if there's any remote divergence in fetched from). That's different a general "merge all these workflows into that", which I don't think does any history checking.

I need to think about this more, but it's an extension of the merge mode thing I added for sandboxes/replace

[josephjclark]

Struggled a bit with version history. Aside from a bug in lightning, I've discovered that checking for divergence on fetch is impossible in the CLI unless we always include the whole version history without squashing. Because if a version is squashed, the CLI can't compare it's local project with the remote one to decide if there's been a divergence.

I panicked for minute but the answer here is: why is fetch checking for divergence? Fetch should just download the latest version and save it to the local project.yaml file. This works on the assumption that project.yaml is never edited by a human. Because that's what the project file is - just like the old state file, it's just as snapshot of the server we pulled from. It should never be modified by the user.

Where we do need a divergence test is checkout. But here, local history is sufficient. On check out (whatever we're checking out from), if our active version has different to the forked_from version, it means we've made local changes that could be lose by checkout. So checkout must fail (unless forced)

I need to ensure there's a really good doc resource that details all this version and state theory. Because it's complex and subtle. It works great when you don't think about it - but any new user coming in to dev this stuff needs to really understand it. A good usecase for AI docs.

[josephjclark]

So I think all that's left here is to update deploy to use the only-merge-new flag, plus whatever tests are needed.