GitHub Actions Is Slowly Killing Your Engineering Team - Ian Duncan
(www.iankduncan.com)
from mesamunefire@piefed.social to programming@programming.dev on 27 Mar 23:25
https://piefed.social/c/programming/p/1926209/github-actions-is-slowly-killing-your-engineering-team-ian-duncan
from mesamunefire@piefed.social to programming@programming.dev on 27 Mar 23:25
https://piefed.social/c/programming/p/1926209/github-actions-is-slowly-killing-your-engineering-team-ian-duncan
Why GitHub Actions is the Internet Explorer of CI, and why Buildkite offers a better path forward for teams that care about developer experience.
#programming
threaded - newest
Im not sure about the CI solution they are recommending, but I do agree there are some sharp edges to GH Actions. Its not the greatest at logs and had hidden issues all over the place.
TLDR; people are idiots. Dependencies are stealing your data. Write your own CI and stop being a lazy fuck.
Tell me you’re unemployed without telling me you’re unemployed
Tell me you’re not a developer without telling me you’re not a developer.
Well, I’m in that Nix shop category. For example, I run tests for my OSS project using lots of Python versions. There’s no need to use a Github Actions matrix for that - Nix does it more efficiently. Doesn’t even require containers or VMs. And the whole setup runs exactly the same locally, on Github Actions, or on Garnix. (But Garnix has better Nix caching than other CI runners, so that’s what I use.)
What’s your solution to handle secrets locally?
Sounds about right.
I’m using GitHub actions at work because this place is extremely dysfunctional, and I can just add GitHub actions without it being a whole “research spike planning meeting impact analysis” six week journey.
I took it from “there are absolutely no checks and Bob broke the environment because he pushed up a change that’s just invalid syntax” to… well, I couldn’t make it block the build on failures but at least now when Bob breaks it again I can point to the big red X and ask why he merged with an error.
If you can get the permissions to do it set the protected branch and prevent merge on build failures. This is a repo setting that will work with any ci provider assuming it updates the PR status.
I thought about it but people are so sensitive here. If they broke something and couldn’t merge they’d probably raise a big stink, and then there’s good odds the checks would be removed “because they’re adding friction” or some nonsense. My boss has already warned me about staying in my lane.
These people have never done any automated testing of any sort. No linter. No unit tests. And they don’t seem to want to.
Ironically, I am one of these people building get up action CI pipelines that automate large multi-service multi-team deployments across different cloud vendors and regions.
It’s all a bit of a pain in the ass and. Actions don’t really provide many of the nice controls and safety guarantees we would want.
However, GitHub actions is relatively straightforward to implement and relatively easy to operate.
It’s just very easy for you to foot gun
I agree with everything he said, but I’m currently living in the simple, honest truth of God’s own “bash” just like he describes and I’m loving it. Maybe someday I won’t. Maybe someday will be soon. Maybe my bash scripts are horrible nightmare fuel. But they’re also my children. I love them. Even the ugly ones.
I do indeed “have 800 lines of bash that reimplements job parallelism with wait and PID files, has its own retry logic built on a for loop and sleep, and parses its own output to determine success or failure.” I do suspect the script is self-aware. This pleases me. I will bend to its desires. If there comes a time when it no longer desires to perform CI for me, I will respect its wishes.
Bash that can do the same job regardless of ci tools is underrated. I likevthat setup too because if your tesm eber needs to run a job/deploy/ect they can with the same exact processes the ci employs.
Why not use python at that point? Sounds like the bus factor would be pretty big on this one
This the most logical solution. Python is what you (should) turn to when your Bash scripts start looking like real software. Whenever I see a (non-geriatric) developer with a giant bash script, my first assumption is that this is a junior dev who doesn’t know better.
My work uses Azure Dev Ops because it’s “free”, I assume it’s effectively just GitHub actions, but without the AI (because no one has updated it in the last 5 years)?
If so, I can agree the agents are awful.
I’ve the impression that it’s moving the problem to another vendor instead of solving it. Not exactly sure how to feel on this topic.
Yeah, I always plead for as much as possible to be automated offline. Ideally, I’d like the CI/CD job to trigger just one command, which is what you’d trigger offline as well.
In practice, that doesn’t always work out. Because the runners aren’t insanely beefy, you need to split up your tasks into multiple jobs, so that they can be put onto multiple runners.
And that means you need to trigger multiple partial commands and need additional logic in the CI/CD to download any previous artifacts and upload the results.
It also means you can restart intermediate jobs.
But yeah, I do often wonder whether that’s really worth the added complexity…
What I usually push for is that every CI task either sets up the environment or executes that one command™ for that task. For example, that command can be
uv run ruff checkorcargo fmt --all – --checkor whatever.Where the CI-runs-one-script-only (or no-CI) approach falls apart for me is when you want to have a deployment pipeline. It’s usually best not to have deployment secrets stored in any dev machine, so a good place to keep them is in your CI configs (and all major platforms support secrets stored with an environment, variable groups, etc). Of course, I’m referring here to work on a larger team, where permission to deploy needs to be transferrable, but you don’t really want to be rotating deployment secrets all the time either. This means you’re running code in the pipeline that you can’t run locally in order to deploy it.
It also doesn’t work well when you build for multiple platforms. For example, I have Rust projects that build and test on Windows, MacOS, and Linux which is only possible by running those on multiple runners (each on a different OS and, in MacOS’s case, CPU architecture).
The compromise of one-script-per-task can usually work even in these situations, from my experience. You still get to use things like GitHub’s matrix, for example, to run multiple runners in parallel. It just means you have different commands for different things now.
Yep. Been there done that. Still doing that. Most of these can be worked around. Doesnt make if less annoying, more infact. But still… its fine.
At the very least its version controlled which is more praise than I can give Jenkins at least one out of the box.
I mean, sure… But for most users of GitHub actions “it’s easy and free” is a huge benefit, easily outweighing any technical advantages Buildkite might have.
I’m not sue who Ian Duncan is but i bet they sell or are in the business of developer tools. They always think their better solution is necessary rather than the enemy of having something is wat better than nothing.
Get off github but action on if it keeps your core compiling and your tests run!
Lot of complaining for issues that are easily side stepped, but maybe cos I’ve learnt it all already, but it does fill the article with lots of words.
The byo compute and logs thing is half the rant which has never even been a problem for me in the last 4 years on GH.
Not sure between config and programming? You literally can choose that yourself.
Biggest issue for CI for me is how to get CI secrets locally… which I don’t know a good answer for.
Nix is great but try getting your whole team / org on it…