CI build artifacts on master have wrong version

[BsAtHome]

The version of the CI builds of the Debian packages is using 2.10.0~pre0. The VERSION file was changed to 2.10.0~pre1 in ac537d1 but the tag was never pushed.
@andypugh maybe you forgot to add --tags to a git push?

(note: when the HAL changes to 64-bit are integrated, then we need a change in VERSION tag again to distinguish history.)

[hdiethelm]

Similar issue: #4030

[BsAtHome]

Yes, indeed. The issue you mention makes it clear that the VERSION file should be the determining factor. But, there is also the git commit hash usually appended. How will that be handled?

BTW, willing to submit a patch so we can discuss our options by example?

[hdiethelm]

I'm looking into it a bit deeper just now. There is a hidden workflow in these scripts. That's why I created an issue instead of just creating a PR. I'm still struggling a bit to make sense of the logic in these and chances are high something will break when changing anything in there.

Right now, the workflow it is the following, right?

Release:
Looking at: 86cdca7

1. Manually: debian/update-dch-from-git creates the changelog -> linuxcnc (1:2.9.10) with all the changes extracted from the git history + some manual adjustments
2. Manually update VERSION
3. Commit
4. Create a tag, let's say 2.9.11, push: CI runs
5. debian/update-dch-from-git does not update the changelog due to it detects there are no new changes.
6. scripts/get-version-from-git writes v2.9.11 to VERSION (Redundant, unchecked)
7. Packages are built using this version

This is a bit dangerous due to the CI basically will update the changelog if any check does not work and the resulting version will be the tag, even if the content of VERSION was committed differently.

Every non-tagged commit:

1. debian/update-dch-from-git updates the changelog with a tag guessed in scripts/githelper.sh and a version based on this tag is created in scripts/get-version-from-git with the commit hash. The logic that breaks on branches based of master due to the newest tag does not mean the highest version number, at least right now.
2. scripts/get-version-from-git writes to VERSION based on ^^
3. Packages are built using this version

Just always using the VERSION file would result in the ability to tag for example 2.9.11 but if the VERSION file is 2.9.10, 2.9.10 will be the resulting package in the CI, so checks are needed.

I also see some checks if tags are signed or not. Are signed tags used right now?

The biggest issue with this "search the last matching tag" is that git has a hard time traversing the history to find the last matching tag on the the history of a random branch. Depending if 2.9 was merged or not, and the tag on 2.9 is newer than on master, it lands on the 2.9 branch. scripts/githelper.sh looks like a try to fix this issue but still fails in some cases.

Finding the right last tag in all cases looks between really difficult to technically impossible and it still doesn't solve the issue that you can change the VERSION file without an effect of the artifacts. So I would suggest a simpler logic:

If tagged:

• Check if VERSION matches the tag, otherwise fail packaging: Prevents having VERSION vs. last tag inconsistency
• Check version in debian/changelog. If wrong, fail packaging: Prevents inconsistent package
• Otherwise build

If not tagged:

• Use VERSION from file and amend with git hash of actual commit: This guarantees that the right base version is used
• Add an empty entry in debian/changelog with the correct version but nothing else. So at least it is not wrong based on guessing a base tag.
  • debian/changelog could possibly be filled by all commits since tag '$(cat VERSION)' if it exists but I don't fully see a value of an auto-generated changelog for master / branches dev builds, it's nearly 10k lines long right now, no one will read it. The git hash in the version should do the job, you can use git tools to check the history if needed.

For changelog:

• Based on debian/update-dch-from-git: Create scripts/create_linuxcnc_release that takes the last tag and actual version as an argument, so no need to auto-detect anything.
  • This script updates debian/changelog and VERSION
  • Now you need to add / tag / commit
• Or just get rid of this script and update VERSION and changelog manually and using git tools. Consistency of tag / VERSION / changelog will be checked in CI on release builds.

This would boil down to 1-2 simple scripts, one that checks the consistency in releases and optionally one to prepare releases.

[BsAtHome]

Very nice analysis.

The build tagged vs non-tagged seems the best solution. You can only build a "real" package from a checkout that matches the tag and VERSION. Otherwise, the VERSION file is the determinant with added git hash, easy, simple and robust.

The changelog,... I think no build procedure should touch the VERSION file. The VERSION file is set to mark a release or intermediate release, regardless whether tagged or not. The commit that updates the VERSION file is the one that could/should be tagged. To do a "full" release with updated changelog needs a document that outlines the exact procedure with explanation and commands to execute. Some scripting may help, but the doc should describe what is needed. The procedure must be followed to have things done consistently.

[hdiethelm]

Ok, so I will create a PR in the next few days. It should also be easy to back port to 2.9 if desired.

Final testing will be a bit cumbersome. Probably I can tag a test release in my own gitlab. The scripts can easily be tested locally, I did this already for the analysis. But the final test will just be the next real LinuxCNC release.

For the changelog: This script would purely be a local helper for people creating releases, not run in CI. You run it on your PC, add the changes, verify, tag, run the consistency check script, commit & push.

Is there already any doc concerning the release process? Otherwise I will add a page in the Developer Manual section.

@andypugh Any input? Looks like you do the releases normally, so it will mostly affect you. However, based on the above concept, nothing will really change for releases.

@BsAtHome Right now, dev builds have the version v2.10.0-pre0-7404-g9fb2ef1e77 where the number 7404 is the commit count since tag v2.10.0-pre0. This number is mostly linearly increasing based on the history, so a higher number means a newer package, except for branches where there will be edge cases, 7404 can exist on a branch and on master but not be the same.
To get this number, the base commit of the last release has to be found. This can be done based on tags or based on commits changing VERSION. Do you think this is needed?
Way simpler would be to just leave it away, so v2.10.0-pre0-g9fb2ef1e77 but then you loose the (not 100% reliable) information which package is newer and the package manager could complain about downgrading.

[BsAtHome]

For the changelog: This script would purely be a local helper for people creating releases, not run in CI. You run it on your PC, add the changes, verify, tag, run the consistency check script, commit & push.

Yes, it should be straight forward, until it isn't... Every time you remember things differently until up is down and left is right. ;-)

Is there already any doc concerning the release process? Otherwise I will add a page in the Developer Manual section.

Not that I know of. Andy mentioned some time ago he had a "complex setup" with quite a bit of "manual intervention" and it seemed cumbersome. Especially remembering the steps ;-)
Making that process streamlined would be great.

Right now, dev builds have the version v2.10.0-pre0-7404-g9fb2ef1e77 where the number 7404 is the commit count since tag v2.10.0-pre0. [snip]

The real problem happens when the VERSION file disagrees with the tags. That is our current problem, there is no tag v2.10.0~pre1. If there is to be a tracking number, which I think is a good thing, then it must be based in the last commit of the VERSION file. That should coincide with a tag, but there are no guarantees. So the count should be commits ahead of the VERSION file.

FWIW, if you use these CI packages, then you are prepared to deal with some fallout. But, for future releases (also in the 2.10 branch), it is probably best to use the count to distinguish pre-release packages from the real deal.

BTW, the 2.9 branch packages seem to use the format linux-uspace_${VERSION}.${COUNT}.${HASH}_amd64.deb (dots in separation instead of dash).

I think,... getting the count is a question of getting the git rev-list --count $(git log -1 --format='%H' VERSION)..HEAD if you just count commits after the VERSION file update.

[hdiethelm]

For the changelog: This script would purely be a local helper for people creating releases, not run in CI. You run it on your PC, add the changes, verify, tag, run the consistency check script, commit & push.

Yes, it should be straight forward, until it isn't... Every time you remember things differently until up is down and left is right. ;-)

At least it needs some doc. Just remembering never works out after some time... ;-)

Is there already any doc concerning the release process? Otherwise I will add a page in the Developer Manual section.

Not that I know of. Andy mentioned some time ago he had a "complex setup" with quite a bit of "manual intervention" and it seemed cumbersome. Especially remembering the steps ;-) Making that process streamlined would be great.

Right now, dev builds have the version v2.10.0-pre0-7404-g9fb2ef1e77 where the number 7404 is the commit count since tag v2.10.0-pre0. [snip]

The real problem happens when the VERSION file disagrees with the tags. That is our current problem, there is no tag v2.10.0~pre1. If there is to be a tracking number, which I think is a good thing, then it must be based in the last commit of the VERSION file. That should coincide with a tag, but there are no guarantees. So the count should be commits ahead of the VERSION file.

FWIW, if you use these CI packages, then you are prepared to deal with some fallout. But, for future releases (also in the 2.10 branch), it is probably best to use the count to distinguish pre-release packages from the real deal.

BTW, the 2.9 branch packages seem to use the format linux-uspace_${VERSION}.${COUNT}.${HASH}_amd64.deb (dots in separation instead of dash).

Tag vs. VERSION vs. Debian package version have conversions between:

linuxcnc/debian/update-dch-from-git

Line 21 in 41568b4

NEW_DEB_VERSION=$(echo "$GIT_VERSION" | sed -r 's/-pre/~pre/; s/-/./g')

and

linuxcnc/.github/scripts/build-package-arch.sh

Line 8 in 41568b4

scripts/get-version-from-git | sed -re 's/^v(.*)$/\1/' >| VERSION; cat VERSION

I think,... getting the count is a question of getting the git rev-list --count $(git log -1 --format='%H' VERSION)..HEAD if you just count commits after the VERSION file update.

If you checkout b71977a, git log -1 --format='%H' VERSION shows the merge commit from 2.9 (b6513d0) due to a newline was changed. Not really what you want but a bit an edge case.

On b71977a, you can easily show the issue:
git rev-list --count $(git log -1 --format='%H' VERSION)..HEAD -> 492 (2.9 merge)
git rev-list --count v$(cat VERSION | tr '~' '-')..HEAD -> 5423 (since tag v2.10.0-pre0)

As an alternative, it could might be enforced in CI that a change in the VERSION file is only allowed together with a tag. But this will create confusion, the CI will fail if you change VERSION without a tag but on master, you can just miss it if you don't check the CI and then others will complain. This probably creates more issues than it solves.

So I would suggest either accept a not 100% reliable COUNT or just leave it away to not make things to complex.

Might be there is an option to track the last real version change in a more or less save way. Traversing the history could land on a wrong branch, so some logic based on the change / content of the VERSION file would be needed to check if the base commit is the correct one.

[BsAtHome]

The 2.9 merge should never have touched the VERSION file. That is a clear merge mistake. The VERSION file is unique for the branch. The VERSION file should always be pushed as a single commit to the branch in question (problem, see below) when the version changes.

The problem we are in at this moment is the missing tag that matches the VERSION file. I think that should be remembered, regardless. Then your second trick will work. However, that also means that you should be able to both push to master and push the tags in one go. That is no longer the case because PRs are now enforced. That also means no one can easily push a single commit of the VERSION file (needs a PR that gets squashed afaik).

Using only the VERSION file commit would be consistent if people keep their hands off of the file for unrelated stuff. Actually, I think we need a warning, somewhere, somehow, that if the tag and VERSION file do not match, then that is visible. Maybe the CI package build should fail when that happens. That means, if your two count methods would result in different counts, then that is an error.

This needs to be signed off by the one doing the releases (@andypugh) and we could use some input from other people aswell. My take is that strictly adhering to a procedure or an error is produced would make things consistent. It may be harsh, but consistency is important. That also means that a commit like the 2.9 merge would have to be undone.

[hdiethelm]

So, first prototype: #4584

The 2.9 merge should never have touched the VERSION file. That is a clear merge mistake. The VERSION file is unique for the branch. The VERSION file should always be pushed as a single commit to the branch in question (problem, see below) when the version changes.

"should have" is always the same. Most people probably don't expect a newline change to be an issue.

The problem we are in at this moment is the missing tag that matches the VERSION file. I think that should be remembered, regardless. Then your second trick will work. However, that also means that you should be able to both push to master and push the tags in one go. That is no longer the case because PRs are now enforced. That also means no one can easily push a single commit of the VERSION file (needs a PR that gets squashed afaik).

Right now, I do the version based on the tag and if there is none, based on changes in the VERSION_BASE file. I could fail the CI if there is no tag but that could result in the master CI failing for multiple days if someone pushes a VERSION_BASE change, not tagging it and no one with the authority to tag is around.

The way the CI works, PR's should be ok: First, you push the commit to a PR, then merge it to master, a package will built with 2.10.0-pre2-0-g161d9a34a6. Then you push the tag, a package will be released with 2.10.0-pre2.

But you could have troubles when you tag the commit in the PR, then you could have a release on a branch. I need to check the github doc and try to test this under my account.

Using only the VERSION file commit would be consistent if people keep their hands off of the file for unrelated stuff. Actually, I think we need a warning, somewhere, somehow, that if the tag and VERSION file do not match, then that is visible. Maybe the CI package build should fail when that happens. That means, if your two count methods would result in different counts, then that is an error.

Miss-match error tag vs VERSION_BASE is implemented, I still need to check all error handling if it bubbles up to the CI.

However, untagged VERSION_BASE change is more difficult. This would enforce first pushing the tag and afterwards the commit. This works in gitlab but I need to check github.

This needs to be signed off by the one doing the releases (@andypugh) and we could use some input from other people aswell. My take is that strictly adhering to a procedure or an error is produced would make things consistent. It may be harsh, but consistency is important. That also means that a commit like the 2.9 merge would have to be undone.

Agreed, better enforce consistency if possible.