What do you check first when a CI pipeline passes locally but fails in CI?
from indidev@lemmy.world to programming@programming.dev on 13 Jul 11:29
https://lemmy.world/post/49385996

Been dealing with this more often lately. Tests pass on my machine, I push, and CI blows up. Usually it’s one of these:

My current debugging flow is pretty basic: check the logs, compare versions, run the exact same Docker image locally if I can. But it still eats 20-30 minutes each time before I figure out the actual problem.

Anyone have a more systematic approach? Like a quick checklist you run through before you even look at the logs?

Also curious — do you replicate your CI environment locally with something like act (for GitHub Actions) or just trust the remote runner?

#programming

threaded - newest

CameronDev@programming.dev on 13 Jul 11:49 next collapse

Lately, we’ve been use Justfiles to setup the env and run the tests. The ci just runs the just recipes.

Makes replicating on the dev machine much easier.

github.com/casey/just

theherk@lemmy.world on 13 Jul 13:52 collapse

I use mise-en-place for this same sort of thing. Many of my shared workflows demand a certain contract be satisfied including things like providing artifact coordinates, mise tasks (build, publish, etc.). One can also use mise to specify the runtime versions to simplify local build and test and ci using the same runtimes.

Of course, there is no best solution, since this is a “how long is a piece of string” sort of question.

atzanteol@sh.itjust.works on 13 Jul 12:33 next collapse

Whatever the logs indicate is the issue? How can there be a fixed response to this?

You should definitely use your builder container to run your tests locally before pushing changes though. That’s one of the advantages of containers - a consistent environment.

Kissaki@programming.dev on 13 Jul 13:15 next collapse

Usually, when CI fails, it’s a flaky CI failure or that I have in fact not run it locally.

I guess some context is missing. My build flow seems much simpler. I check the CI steps overview and step log, and then I know what’s wrong. That doesn’t take 20 min. More like ~3 min.

At work, we use Jenkins and the runners are owned infrastructure. If I debug what goes wrong in the CI environment, I go into the pipeline definition, and do the calls locally, just like that, on the command line. No need for complicated environment replication beyond that. dotnet restore, dotnet build --no-restore, dotnet package --no-build, dotnet test --no-build, etc.

If it’s not CI specific, issues show up in the normal local tooling without special env prep.

red_bull_of_juarez@lemmy.dbzer0.com on 13 Jul 13:17 next collapse

How viable moving over to carpentry or metal working would be.

eager_eagle@lemmy.world on 13 Jul 13:20 next collapse

do you replicate your CI environment locally with something like act (for GitHub Actions)

this is what does it for me most times; but I usually also have a CI .env file act can use and use just to abstract recipes e.g. when running just test, either CI or local will run depending if the CI env var is set. It’s the same battery of tests, only different env files.

tal@lemmy.today on 13 Jul 14:00 next collapse

If you have a known good (local) and known bad (remote), the first thing I’d probably do is run diff on the local and remote logs. Use a regex or something to strip timestamps or similar that will always differ.

If your tests take 20–30 minutes to run, I think that I’d look into having a way to run a subset of the tests, so you can just run the minimal amount to do the failing one.

If you have a lot of environment problems, I think I’d focus on getting an automated build of the environment to a standard state. If that takes too long, setting up caching or whatever of downloaded packages.

stsquad@lemmy.ml on 13 Jul 16:42 next collapse

In QEMU all of our CI environments are replicable locally as docker/podman images. However usually flaky CI is due to races exposed on overloaded runners so I often also run make -j(nproc) at the same time to simulate that. A retry script is also useful to get an idea of how stable a test is. Having sanitizer builds can also help.

hallettj@leminal.space on 13 Jul 23:00 collapse

I try to capture every detail of the build and test environments in Nix devshells. And where I can I try to encapsulate as much as possible in Nix checks and packages which run in build sandboxes - both locally and on the server. Build sandboxes don’t work for everything, but the devshells alone are great for reproducibility.

  • Wrong interpreter version? A devshell with a flake.lock file ensures every environment is using the exact same interpreter.
  • Accidentally picking up stuff from the local .env? Sandboxed checks and builds don’t get any files that aren’t version controlled, so that’s not an issue. But it’s still an issue with devshells.
  • Accidentally picking up programs or env vars in your environment? Sandboxed builds always get a clean starting environment. If you run nix develop --ignore-env you get a devshell that also gets a clean starting state.

Nix doesn’t fix everything.

  • File system case sensitivity - depending on where this issue presents (program-generated files vs source files), I use property testing to catch this problem. In fact I was working on exactly that the other day.
  • Timing issues - that’s a good old fashioned hard problem. Try to make logical dependencies explicit. It’s really easy to get implicit order dependencies in concurrent code if you aren’t on guard. In languages that support it promises or futures are good for spelling out what needs to happen in what order.
  • Edit: Difficulty keeping secrets in sync - one option is to use Sops or Age to put encrypted secrets in version control. Then your CI only needs to be configured with one secret to decrypt the other secrets it needs.