I've been struggling with a rather complex shell script, and it's becoming apparent that Bash might not be the best choice for this particular task. While I usually gravitate towards statically typed languages like Go or Rust, I've noticed that many people recommend alternative languages such as Lua or Python for scripting tasks.
I'm curious to know your opinions and experiences with scripting languages for larger or more intricate shell scripts. Have you ever encountered a situation where Bash just didn't cut it, and if so, which scripting languages did you turn to for a more effective solution? Are there any specific languages you found particularly suitable for debugging, testing, or handling complex logic in your shell scripts?
For years my go to (after Bash) was Python. However, in the last few years I’ve switched to Rust for any kind of shell command wrapper or CLI tool.
TL;DR I think Rust is best suited to more complex CLI work.
Exact same story here. Bash -> Python -> Rust.
Generally speaking, people should settle on a compiled language if they can. They can iterate as fast as interpreted languages these days.
Edit: If you want to try something different in scripting, try the execline language. Its interpreter processes the script and exits immediately even before the script execution begins. Traditional shell interpreters (like bash) stay active till the entire script is finished. Execline achieves this by a clever chaining of Unix execs, forks and variable substitutions. This makes execline scripts lighter (useful in embedded systems), more secure and less error-prone than traditional scripts. The downside is that writing them will feel a bit weird - since the fundamental paradigm is different from regular shells. However, that will be a refreshing change if you're someone who likes to experiment and try new things.
Yes, and Rust with incremental compilation is pretty fast to iterate as well, as long as you don't use massive libraries/build-scripts etc.
Would you have any example (not necessarily yours) to showcase this? I mean, how is it better suited than say, C++?
Not op, but I feel the same as them.
Compared to C++, Rust has a very good toolchain and libraries. With C++ setting up a project that has dependencies is... painful. I'm a full-time C++ programmer with over 8 years of experience and if I didn't have to, I would never choose it for something new.
With Rust creating a new project and adding dependencies is trivial. There are a lot of great libraries and the ease with which you can use them is very empowering.
Clap and serde are super powers for CLI programs 😀For smaller scripts that don't yet "deserve" full rust treatment, I now use nushell for personal projects.
I too use Rust for what normal people use shell scripts for. But I have a feeling that Rust is falling into the same trap that other languages with similar easy dependency management fall into (Python and NPM are good examples). You end up with a dozen direct dependencies and hundreds of indirect ones with dozens of levels of hierarchy. C and C++ programs have fewer dependencies because each additional one adds more headache for the developer. Drew Devault's Hare language is giving language repo and package manager a skip for the same reasons. And I'm starting to think that he may have a point.
I think it's not that bad yet, when comparing with npm. Usually the dependencies I use are of very high quality. But I'm also very selective with dependencies. I'm rather writing a simple part myself, than using a not-really maintained low-quality dependency...
Btw. I have not looked into the Hare language yet (will do that now), but if it's similar as deno, I won't like it. You want to have some kind of package management IME...
Drew Devault’s Hare language
Ok, they say "use your distros package-manager", that's basically asking for the same disaster as C or C++. I think cargo is one of the selling points of Rust.
At least say something like we use "Nix" for default package-management (which does a lot of things right)...
I personally don’t have any real experience with Go. Lots of smart folks I work with love it. In general, most of what I have read suggests that Rust is better suited to CLI tooling. For my use case it came down to:
- Rust’s cargo system
- The clap crate (which supports building out bash shell completion scrips via a Rust build script). Basically means I can generate a completion script at compile time and include this in the package I distribute to users)
- Rust’s out of the box performance
- The heavy lifting done by the borrow checker in bringing safety
Just curious have you tried Go for this? Go was recently approved at work and I have seen articles about Go for things like this and just wondering if it is worth it. I have been using ansible and chef but need to explore other options. I want to use Rust but I know the road blocks I will have to work through at work. So just wondering if you had any insights to Go over Rust
It's especially true when you want to parse some json/xml/whatever. Just describe your datastuctures with regular struct and enum, add serde and done! It's like magic!
I personally don’t have any real experience with Go. Lots of smart folks I work with love it. In general, most of what I have read suggests that Rust is better suited to CLI tooling. For my use case it came down to:
- Rust’s cargo system
- The clap crate (which supports building out bash shell completion scrips via a Rust build script. Basically means I can generate a completion script at compile time and include this in the package I distribute to users)
- Rust’s out of the box performance
- The heavy lifting done by the borrow checker in bringing safety
I was in a similar situation not too long ago.
My criteria for another scripting language included that it should be preinstalled on all target systems (i. e. Debian and Fedora), it should be an interpreted language and it needs to have type safety.
Afterall I settled with Python due to its popularity, its syntax and features (type safety since v3.6, etc.) and the fact that it is preinstalled on many Linux distributions. System components often use Python as well, which means that libraries to interact with the system tend to be included by default.
for larger or more intricate shell scripts
Those are call applications. Use any language you like. If go/rust is what you know use them. I use rust all the time for things beyond run a bunch of commands and tends to be my go to when I need to process data in any way.
Look at Raku. This seems to be where Raku shines relatively brightly.
pyinvoke.
You can create quick and dirty CLIs, invoke shell commands, and have all of python available for things like parsing config files, getting and setting environment variables, and making remote REST calls.
Totally second this.
But I 'd like to recommend starting with subprocess module which is built-in. Then go ahead and see what the extensions have to offer... There are other that may come handy, like Typer from Tiangolo, and Fire from Google.
For raster graphics image processing, I'd highly recommend G'MIC. Otherwise, Python and especially for string using regex library. I wish there was a vector graphics version of G'MIC.
I've used C# in the past. There's a tool called dotnet-script that lets you run csharp files as scripts without having to compile.
It does require you to install dotnet though.
But tbh it's pretty easy to just chuck an executable together.
Really surprised no one has mentioned Ruby. It's installed by default on almost every system out there (unlike python), it will have the same features on every platform (unlike python where you might get 2.7 or 3.x depending. It's simple and easy to read, and only slightly more verbose than bash. It's very well suited for scripting (please don't use it for application work). It also took a lot of its design from Perl, which a bunch of people are mentioning in this thread, and as a result has a ton of the features of perl, along with a ton of features from other languages. Rust is heavily based on Ruby's design, and i've used Rust to create cli programs and I wouldn't recommend it. It's good, but most cli programs don't need the difficulty of rust for the benefits that rust gives.
Anyway, python has a really really good cli library called Click, but that's about the only good thing about it. If you are looking to use this script on multiple systems then Ruby will be much easier to transfer between systems (it will just work). I've deployed complex python, rust, and ruby CLIs across an org and Ruby was the only easy one. Rust was second easiest, Python absolutely terrible.
If you're not deploying this to other platforms or sharing it across a team or something like that then a lot of the downsides and upsides here don't really matter. Just use the easiest language.
Aw man, you can't write all that and then not give an example!
Ruby makes scripting drop-dead simple. You can run any shell command by surrounding it with back ticks.
# simple example, just grab files: files = `ls`.split("\n") # pipes work inside back ticks files.map {|f| `cat #{f} | grep "can I use grep w/out cat"`} .compact .each { |match| puts match } # easy to build a pipeline on the data in ruby, too!That's it! No messing around with popen3, or figuring out pipes or signals. Those are there too if you really need them, but if you just wanna write a quick script with a less arcane syntax - try Ruby!
Haha sorry, I wrote it all on my phone while traveling. and yeah, if you're running just shell commands it looks almost the exact same as a bash script, and then when you need actual scripting capabilities you get them.
Rust is heavily based on Ruby’s design
I would not say "heavily based". Literally only the closure/lambda syntax, which is cosmetic. Rust is mainly inspired by ML-family languages and C++.
I think Ruby is a reasonable choice for small scripts which someone might otherwise use Python for. But Rust is very well suited to more complicated or long-lasting command-line tools, especially if performance is at all a concern. Clap alone is super nice, but there are a lot of awesome libraries for making rich CLI tools easily.
- colored or owo-colors for adding terminal colors super easily
- crossterm for cross-platform terminal feature support
- rustyline for all your fancy readline needs
- cursive for TUIs
- comfy-table for command-line tables
And like....a hundred more I could mention. Idk, for anything that's not completely trivial, which will be used and maintained by humans and not thrown away, Rust is really nice.
I would not say “heavily based”. Literally only the closure/lambda syntax, which is cosmetic. Rust is mainly inspired by ML-family languages and C++.
All of Cargo is based on (and created by) the same person that created bundler for ruby. That list also misses out on a lot of things, like
!, automatic returns, honestly most of the actual language 'design' rather than the internals (that seems to be a list of where the architects got their ideas for internal implementation as well, rather than just the readability of the language).But Rust is very well suited to more complicated or long-lasting command-line tools, especially if performance is at all a concern. Clap alone is super nice, but there are a lot of awesome libraries for making rich CLI tools easily.
I disagree. Like I said, I wrote command line apps in all of these, performance was a factor. But a much larger factor is getting other devs on your team to contribute. And that was just absolutely impossible with Rust. The learning curve is just too high. For something that isn't a hobby project, but that you might need a team member to roll out a fix in just a few hours, Rust will not cut it.
Yes, you will have way more bugs in all the other programs, but honestly I had a shit ton of bugs in my rust cli as well, because, it turns out, rust works really well when it has control over everything, but man does it suffer when you have to interface with the real world.. And oh boy did that make it incredibly difficult to write. Like I said, I deployed CLIs in all three of these languages. Ruby was the easiest of them all. Not just in development, but also maintenance.
As an android developer working with GitHub actions I just use Kotlin Script: but that's because I'm already writing in Kotlin all the time.
It's really nice to be able to run it as a scripting language though
