Lately I've been writing some new ("
I feel a bit silly about this: isn't
Rust what you're supposed to use
if you want metal-level performance and are unconstrained by past
engineering decisions? Or
Something on the
C++? Why use
a 50 year-old language that is notorious for memory unsafety and
There are a few things about this situation that are a pretty good fit
I'm prototyping in a data-heavy situation, iterating on the
right way to solve various problems. Normally I would use Python for prototyping, and I am
using it a bunch here. In many cases, however, I need to run my code
quickly over a large amount of data and see how it works in practice,
and my Python implementations have generally been far too slow.
I'm implementing algorithms that are computationally
straight-forward: a Trie to count unique
fixed-length substrings (code),
or approximating the previous algorithm with multiple processes writing
to a big block of shared memory and accepting collisions (code).
If I needed library support or had a large amount of tricky logic I'd
use a different tool.
I'm processing very simple
formats: essentially just long strings of
[ACGT]*. This keeps the code rule
of two compliant. I wouldn't want to use C for any tricky parsing
unless it was sandboxed and I was very confident in the sandboxing
I'm working on this set of problems mostly by myself, so I
should choose the tooling where I'll be able to make progress most
quickly. It doesn't matter much right now if my code is a bit weird.
Once I get a better handle on how we're going to approach this problem
computationally it will likely make sense to rewrite in a modern
language, which will both be safer and more readable.
The biggest risk here is that prototype code will become production
code, there will always be something more urgent than a rewrite, and
at the core of our system we have a chunk of code in an unsafe
language that's poorly documented and confusing to read. This is
probably the strongest argument against starting anything in C, even
for prototyping. While I can't completely commit to ensuring this
doesn't happen, I'm going into this with my eyes open. And I will at
least commit to seriously documenting anything that's becoming
This is a bit of an unusual confluence of factors, and if you'd asked
me a few years ago if I was ever going to write C professionally again,
let alone choose to start something in C, I would have said no. Yet,
in this case, I think it's the right call.
(Large parts of my rhythm stage setup,
including both the MIDI
routing and whistle-controlled
synthesizer are also in C, though there for minimizing latency
instead of maximizing throughput. Since that's something silly
I'm doing for fun I feel less weird about it.)
Comment via: facebook
Probably dumb questions:
- Have you tried PyPy? It might increase the number of cases where python is good enough.
- Why C instead of C++? (I assume Rust would slow down prototyping speed because of the borrow checker). Is it because you're more familiar with it from your side projects?
Have you tried PyPy?
Have you tried PyPy?
I haven't tried PyPy for this particular project, but my experience previously had been that while I usually got a bit of speedup it wasn't typically much.
Why C instead of C++?
Why C instead of C++?
I used to work in C++ and know it pretty well, but I don't really like it very much and there isn't anything in C++ I've needed here.
Thanks for the reply!
That's also my experience in most cases, but in others can be much faster. It does especially well on code with lots of looping that can be JITted.As a data point, https://github.com/jeffkaufman/kmer-egd/blob/main/count-quality.py is ~3x faster on my machine (takes 0.8 seconds vs 3 seconds on 100k lines of length 151 containing FF).Which probably is not enough to make a difference, but might still be useful.
Yes, probably not enough to make a difference; that one in particular is fast enough in python. But useful to have the number!
You can also use Numba to speed up loops. It's still slower than C, but it's much better than plain Python code, and it's really easy to implement (just import numba and put a @numba.njit() before your function).
I'm curious if you looked into any of the "better C" languages that have come out lately - Odin/Jai/Zig.
C with some basic object support (single inheritance, interfaces) and a better/safer string manipulation library (all mutable strings are structs with current/max size, no null-termination) and without implicit new/free of C++ is what I miss in most of my C projects, and have to recreate or import on any new greenfield project. There are probably C frameworks out there that give you that. Goes a long way toward a safer C without performance overhead of managed code.
The way I approach situations like that is to write code in Lua and only push stuff that really has to be fast down to C. (Even C+liblua / using a Lua state just as a calling convention is IMHO often nicer than "plain" C. I can't claim the same for Python...) End result is that most of the code is readable, and usually (i.e. unless I stopped keeping them in sync) the "fast" functions still have a Lua version that permits differential testing.
Fundamentally agree with the C not C++/Rust/... theme though, C is great for this because it doesn't have tons of checks. (And that's coming from someone who's using Coq / dependent types regularly, including at work.) Compilers generally want to see that the code is safe under all possible uses, whereas you only care about the specific constellations that you actually use when prototyping. Convincing the compiler, disabling checks, and/or adding extra boilerplate adds overhead that seriously slows down the process of exploration, which is not something that you want to deal with in that mode.
Why aren't you using the various C compiler options that do all the pointer/array bounds checking that the Rust fan bois think are unique to them?
Plus there are the integer overflow checking options that Rust eventually got around to supporting.
Probably worth doing! Recommendations?
I'm a long-time hardcore bounds-checking fan.
Others prefer: -fsanitize=address,undefined,bounds-strict