This post does not contain the canonical QURI opinion, but I am a contractor there. I want to thank early commenters Vivian Belenky and Ozzie Gooen. The section concerning probabilistic programming is readily skippable to make things easier for nonprogrammers.
In this post, I aim to explain and describe what Squiggle is. In future posts, I will clarify its EA value proposition and highlight its applications and theories of change.
Belief specifications are descriptive units of belief. Estimational programming (EP) is the practice of writing belief specifications. We want EP products to be functional and uncertain. When I say an EP product is uncertain, I mean that it allows uncertain beliefs to be specified. By functional, I mean that when users write estimates on an EP product, the estimates should be reproducible and composable. By composable, I also mean to say that they should be transparent and interpretable. If an estimate is both reproducible and interpretable, I will call it auditable.
Technologies for writing belief specifications exist but lack the above desirable properties. Users can accomplish estimational programming in each of them to varying degrees, but Squiggle is unusual because it is a platform for which EP is the first-class intended use case.
# Hat tip to Jonas Moss for the code
import numpy as np
import scipy.stats as st
import matplotlib.pyplot as plt
rng = np.random.default_rng(313)
n = 10000
# Translate quantiles
a = np.log(0.75)
b = np.log(0.9)
k1 = st.norm.ppf(0.05)
k2 = st.norm.ppf(0.95)
sigma = (b - a) / (k2 - k1)
mean = b - sigma * k2
transfer_efficiency = np.random.lognormal(
x = np.linspace(0.7, 1, 100)
plt.plot(x, st.lognorm.pdf(x/np.exp(mean), sigma)) # Scipy's parameterization of the log-normal is stupid. Cost me another 5 minutes to figure out how to do this one.
### It's prudent to check if I've done the calculations correctly too..
np.quantile(transfer_efficiency, [0.05, 0.95]) # array([0.75052923, 0.90200089])
transfer_efficiency = 0.75 to 0.95
In addition to these gains, in a general purpose programming language, any nontrivial computation forces users to do Monte Carlo longhand. The programming perspective in notebooks is very free; we have trivial/apparent Turing completeness, we can ingest data and do intensive numeric work on it, but the level of generality provided leads to the imposition of boilerplate costs when users want to write belief specs.
It is not enough to identify what estimational programming is; I must also identify what EP ought to be, which is uncertain and functional. Indeed, the term of art may even be “functional estimational programming” or “programming compositional estimation functions'' or variants. Still, I don’t want to be clunky, and I don’t want to separate the aspirations of EP from its history.
I can mostly punt this to Superforecasting and How To Measure Anything for the details, but TLDR, you’re uncertain about everything. Belief specs aren’t handy without being in confidence intervals.
Functionality, or composability, is a desirable property for an estimational programming language.
MicroCOVID dashboards are estimates of the risk involved in doing various activities. However, they can’t export to Guesstimate sheets. It would be helpful to send our covid risk tolerance into an estimate of how much fun we’ll have, value we’ll create, or resources we’ll consume. The reason this might not have seemed thinkable to the developers of either project is that they’re not working from a shared notion of belief spec.
I think a lot about this article by Fabrizio Genovese, which defines compositional systems as fluid de- and re-composition under the abolition of emergent properties, contrasting it with modular systems that, like a house’s electrical wiring, might blow up in your face if you don’t understand how you’ve put it together even if you understand the individual parts. Both are described as “breaking things apart and putting them back together,” which we’d like to do with forecasts and cost-effectiveness analyses. Still, the compositional timbre emphasizes ease of an audit: the parts you break things down into provide understanding, show you how to check somebody’s work, or your own, and the piecing back together is so simple that it is precisely – no more than, no less than – the sum of the parts. I hope this illustrates how I associate interpretability/auditability with compositionality.
This aspiration of compositionality (rather than modularity or black box stories) is a game-changing aspect of forecasting and cost-effectiveness analysis for the following reasons:
git diff Alice/worldview.squiggle Bob/worldview.squiggle
This section, may you be warned, is more technical than other sections.
Some of you have heard of a programming language where the terms are distributions. There are two kinds of literature going by the name probabilistic programming (PP). In one of them, terms are random variables, and we ask questions like “probability of halting” or “expected length of redux chain” (see citation pdf chapters 1 & 2). In the other, the terms are distributions, and we ask questions like “what sampling process approximates the posterior of a dataset and a supplied prior” or “given a sampling process, can we extrapolate integrals and derivatives of the implied density function” (see SR, BDA). I’m pretty sure the former case involving random variables is obscure and too academic, whereas the latter case involving distributions can be used by scientists from diverse departments. This latter case is also what people ask the Squiggle team about all the time. An academic paper properly comparing and contrasting formal properties of estimational and probabilistic programming is somewhere in the “not a priority” or “eventually” region of the QURI roadmap (nevertheless, firstname.lastname@example.org if you have ideas about what that would look like), but I’d like to take a quick pass here.
Now, quickly showing contrast doesn’t on its own justify the existence of Squiggle when existing ecosystems are “nearby” in the usable space. Still, it should help explain the relative value prop of using one or the other for your particular project or use case.
Squiggle is simply the first open-source language that emphasizes EP as a first-class citizen. It is not the sole keeper of what EP is all about or where it’s going. By another token, by putting Squiggle in this box I’ve created, I may be constraining the reader’s imagination about where Squiggle could end up.
I think the transfer_efficiency example is kinda dishonest (also kinda wrong: the right-hand column uses an upper figure of 0.95 but the left-hand column uses 0.9). Squiggle apparently has special-case support for the special case of a lognormal variable where you know the lower and upper fifth-percentile values. That's nice, I guess, but it really is quite a special case.
Unless I am going to be spending all my putative time in Squiggle specifying fifth-percentile values of lognormal variables, that example is of no use to me without knowing what happens if I want to specify the tenth-percentile values instead, what happens if I want a normal distribution, etc.
I had a look in the Squiggle documentation, and it looks to me as if the "to" syntax is only for 5th-to-95th percentiles, and only for normal distributions where those quantities have different signs and lognormal distributions where they have the same sign. (Which means, e.g., that if you are calculating those quantities then you cannot safely use "to" unless you know for sure what the signs will end up being.) If you have Squiggle code that uses "to" and then you realise that actually you wanted to use the 10th and 90th percentiles, or a different distribution, or a normal distribution even though both those quantiles are positive, you need to throw away that one line and write something completely different that does the same untidy thing as those "15 lines" of Python code. Is the corresponding Squiggle code in that case actually any shorter or clearer?
Also, those "15 lines" of Python code corresponding to the "1 line" of Squiggle code include
and if you were in fact writing something in Python that makes substantial use of distributions specified by giving a couple of quantiles, you would write a function containing the boilerplate once and never have to worry about the fiddly details again.
(On the other hand, if you're only doing it once then indeed the boilerplate is a substantial fraction of your code. But if you're only doing it once then I bet the broader advantages of a general-purpose programming language will outweigh the local advantages of a domain-specific language specializing in doing neatly something you only need to do once.)
For the avoidance of doubt, this is not intended to imply that Squiggle is bad. Just that that example seems super-unfair.
[EDITED to add:] I'm also really confused, because the actual Squiggle notebook linked to from the discussion this example supposedly comes from (1) doesn't in fact define transfer_efficiency this way, (2) does consider at least one lognormally distributed quantity determined from estimates of its quantiles, namely consumption_prior, and (3) handles that quantity ... by giving explicit numerical values for the mean and stddev of the underlying normal distribution. There is a quantity in the spreadsheet called transfer_efficiency, it is beta-distributed not lognormal, and its distribution is also specified by giving magic numbers for the distribution parameters rather than saying where two of its quantiles are.
I'd be really interested in a head-to-head comparison with R on a bunch of real-world examples of writing down beliefs that were not selected to favor either R or Squiggle. R because at least in part specifying and manipulating distributions seems to require less boilerplate than in Python.
0.75 to 0.95
0.75 to 0.9
Thanks for the flag!I might not be understanding correctly, but I don't think there's a problem here with the actual underlying code just my explanation of it (we all hate magic numbers). Which is very fair enough, the notebook is much too dense for my liking. It's still a work in progress!
I agree! The Squiggle team is looking to create different quantiles for different distributions. I've needed them on several occasions. You can check out the discussion on GitHub here. It's on my todo list.
Just letting you know that you seem to have double-pasted the 3rd bullet point.
oof, good catch, fixed.
the author re-reading one year+ out:
I haven't been working on this stuff except a little on the side for most of the last year, but still get excited here and there. I returned to this post because I might have another post about module systems in software design, package management, and estimational programming written up in the not too distant future.
Overall, this remains a super underrated area.