An Apprentice Experiment in Python Programming, Part 2

by konstell, gilch15 min read29th Jul 202118 comments

29

ProgrammingApprenticeshipDistillation & PedagogyPractical
Frontpage

[Epistemic status: me trying to recall what happened in a 4.5-hour pair-programming session, during which I was focused on solving problems. This post was reconstructed from scratch code files, written messages (the session was over voice call) and my python shell history.]

Previously: https://www.lesswrong.com/posts/kv3RG7Ax8sgn2eog7/an-apprentice-experiment-in-python-programming

Three days ago gilch and I had another session on Python Programming, where we continued talking about decorators.

Lambda decorator

So far, all of the decorators we've seen have been functions that take in another function as parameter, then used by the syntax @<decorator_name>. Turns out we can use a lambda function as a decorator too:

@lambda f: print("hi")
def add(a, b):
    return a + b

When this file gets run in Python 3.9 interactive mode, the output is

hi

and when I call the add function:

>>> add(1, 2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object is not callable

Gilch asked me to write out the desugared version of this code. I wrote add = print("hi"), which was not correct. Gilch gave me the correct desugared code:

def add(a, b):
    return a + b

add = (lambda f: print("hi"))(add)

Partial

We did a brief introduction to partial. We had code from last time:

def subtract(a, b, c):
    return a - b - c

Accessing doc from within python shell:

>>> from functools import partial
>>> help(partial)

Using partial to create a function with the first and second arguments passed in:

>>> partial(subtract, 1, 2)
functools.partial(<function subtract at 0x7f34649af3a0>, 1, 2)
>>> partial_sub = partial(subtract, 1, 2)

Gilch pointed out that the line above, partial_sub = partial(subtract, 1, 2), could've been written as partial_sub = _ because in python shell, _ returns the last value that's not None.

Testing the partial function:

>>> partial_sub(3)
-4

1 - 2 - 3 equals -4, so we're getting what we expected.

Gilch then pointed out that arguments can be passed into partial as keyword arguments:

>>> partial(subtract, b=0, c=2)
functools.partial(<function subtract at 0x7f34649af3a0>, b=0, c=2)
>>> _(10)
8

Finally, there's a way to implement partial functions using lambda:

>>> lambda x: subtract(x, 0, 2)
<function <lambda> at 0x7f346474ae50>
>>> _(10)
8

Running Bash in Python

While running help(partial) in the python shell, the topic of using less came up. I asked gilch if it was possible to pipe outputs from python shell to bash tools, gilch answered that we could not do it directly, but just like how bash can run python, python can also run bash:

>>> import os
>>> os.system("bash")

Now we're in a bash shell, run by python:

~/code_stuff/python_scratch $ ls | less
~/code_stuff/python_scratch $ exit
exit
0
>>> 

At the last line, we're back in python shell again.

Lambda functions

Gilch gave me a number of lambda expressions in increasing complexity, asked me to explain what they did, then verify:

  • (lambda x: x+x)(3) evaluates to 6, this one was straightforward
  • (lambda x: lambda y: x + y)(2)(3) I assumed that this statement expanded inside out (i.e. 2 was passed in as y and 3 was passed in as x) so I concluded that this would evaluate to 5. I got the correct answer, but my mistake was exposed by the next example:
  • (lambda x: lambda y: x + y)('a')('b') Using the same logic, I concluded that the result would be 'ba', but the actual result was 'ab'. Why did I make this mistake? Gilch gave me the next question:
  • (lambda x: lambda y: x + y)('a') evaluates to a lambda. A lambda of x or y? The first part of the expression, (lambda x: lambda y: x + y), is a lambda function that takes in a parameter x, and returns a lambda function lambda y: x + y. This lambda function that takes in x is then called with argument 'a'. The return value is lambda y: 'a' + y. And then it made sense why the previous question evaluated to 'ab' instead of 'ba'.

I think what gilch was getting at here was the syntax of function calls other than the format of f(x). Gilch also at some point pointed out the difference between a callable and a function.

Preprint

Gilch gave me the next challenge: write a decorator preprint such that it can be used in the following way:

@preprint("hi")
def bob():
    print("Bob")
@preprint("hello")
def alice():
    print("Alice")
>>> bob()
hi
Bob
>>> alice()
hello
Alice

I struggled here. I attempted to write out some vague idea in my head:

def greet_by_name(salutation, name):
    print(salutation)
    print(name)

def bob(salutation):
    from functools import partial
    return partial(greet_by_name, name="Bob")

Gilch reminded me that the use case was already specified; functions alice and bob were not to be modified. They then suggested that I wrote out the desugared version of the code. I struggled and gave an incorrect answer again until gilch provided me with the correct desugaring:

@preprint("hi")
def bob():
    print("Bob")

is equivalent to bob = (preprint("hi"))(bob).

Understanding decorator de-sugaring

As you can see, at this point, I was struggling with desugaring. So gilch presented me with a formula:

@<decorator expression>
def <name>(<args>):
    <body>

is equivalent to

def <name>(<args>):
    <body>

<name> = (<decorator expression>)(<name>)

Preprint

My memory of what exactly happened here is spotty, perhaps because I was focused on finding out how to write the preprint decorator. Eventually I came up with something like this:

def preprint(greetings):
    from functools import partial
    def greet_by_name(greeting, foo):
        def wrapper():
            print(greeting)
            foo()
        return wrapper
    return partial(greet_by_name, greetings)

This worked, but you may be able to see from the way I passed in greetings into greet_by_name, I was confused about why there needed to be 3 layers of definitions. Gilch gave me an alternative solution, and explained the pattern:

def preprint(greetings): # factory
    def greet_by_name(function): # decorator
        def wrapper(): # replaces the function being decorated
            print(greetings)
            return function()
        return wrapper
    return greet_by_name

Gilch explained that, for decorators that take in parameters, they are like a factory that produces different decorators based on input. Thus the first layer of definition processes the input, the second layer of definition is the decorator itself, and the innermost wrapper function is what replaces the function that ends up being decorated. This

@preprint("hi")
def bob():
    print("Bob")
@preprint("hello")
def alice():
    print("Alice")

is equivalent to

greet_by_name_hi = preprint("hi")
greet_by_name_hello = preprint("hello")
@greet_by_name_hi
def bob():
   print("Bob")
@greet_by_name_hello
def alice():
    print("Alice")

At this point, I was not quite getting the 3-layer structure--I identified that there was a pattern of factory-decorator-function but I was mostly mimicking the pattern. This lack of understanding would be exposed in a later example.

wraps

In the previous example, we had the decorator preprint:

def preprint(greetings): # factory
    def greet_by_name(function): # decorator
        def wrapper(): # replaces the function being decorated
            print(greetings)
            return function()
        return wrapper
    return greet_by_name

Gilch instructed me to add a doc string to a function being decorated by this decorator. So we have:

@preprint("hi")
def bob():
    """
    prints 'bob'
    """
    print("bob")

What happens when we try to access the doc string of bob?

>>> help(bob)

Help on function wrapper in module __main__:


wrapper()
(END)

Gilch then introduced wraps. According to the documentation, it calls update_wrapper to reassign some attributes of the wrapped function to the wrapper function.

from functools import wraps
def preprint(greetings): # factory
    def greet_by_name(function): # decorator
        @wraps(function)
        def wrapper(): # replaces the function being decorated
            print(greetings)
            return function()
        return wrapper
    return greet_by_name

@preprint("hi")
def bob():
    """
    prints 'bob'
    """
    print("bob")
>>> help(bob)
Help on function bob in module __main__:

bob()
    prints 'bob'
(END)

map

Gilch asked me if I knew what map in python did. I said map took in a function and an array and return the result of the function applied to each element of the array. Gilch responded that it was mostly correct, with the caveat that map took in an iterable instead of an array.

Gilch gave me a test case for which I needed to implement the decorator:

@<something>
def double(x):
    return x+x
assert list(double) == ['aa','bb','cc','dd']

I came up with

@lambda f: map(f, 'abcd')
def double(x):
    return x+x

When we run assert list(double) == ['aa','bb','cc','dd'], python did not complain. However when we ran list(double) again, we saw an empty array:

>>> list(double)
[]
>>> double
<map object at 0x7f3ad2d17940>
>>> list(double)
[]

Gilch explained that double was not an array, it was an iterable. After we had iterated through double in the assert statement, it became empty.

Callable

Gilch illustrated the difference between a function and a callable:

def my_if(b, then, else_):
   return then if b else else_

in which then if b else else_ is a callable. Two use cases:

>>> my_if(1<2, print('yes'), print('no'))
yes
no
>>> my_if(1<2, lambda:print('yes'), lambda:print('no'))()
yes
>>> my_if(1>2, lambda:print('yes'), lambda:print('no'))()
no

In the first case, print("yes") and print("no") are not callables (they get run and evaluated to None) whereas the lambda:print('yes') and lambda:print('no') in the second case are callables.

Similarly, when we put the print statements in functions, the functions are callables:

def print_yes():
    print("yes")

def print_no():
    print("no")
>>> my_if(1<2, print_yes, print_no)
<function print_yes at 0x7fa36ffbb280>
>>> my_if(1<2, print_yes, print_no)()
yes

Branching, with 1 branch

Gilch illustrated how we could use decorator for branching:

def when(condition):
    def decorator(function):
        if condition:
            return function()
    return decorator
>>> @when(1<2):
... def result():
...     print("Yes.")
...     return 42
... 
Yes.
>>> assert result == 42
>>> @when(1>2):
... def result():
...     print("Yes.")
...     return 42
... 
>>> assert result == None

Branching, with 2 branches

Gilch asked me how I would implement branching such that if a given condition is true, one branch gets evaluated; if condition is false, another branch gets evaluated. This would essentially mean we decorate two functions. I came up with this solution:

def if_(condition):
    def decorator(function):
        if condition:
            return function()
    return decorator

def else_(condition):
    def decorator(function):
        if not condition:
            return function()
    return decorator

condition = 1<2

@if_(condition)
def result():
    print("Yes.")
    return 42

@else_(condition)
def print_no():
    print("no!")
$ python3.9 -i map.py
Yes.
>>> result
42

And with condition = 1>2:

$ python3.9 -i map.py
no!
>>> result
>>>

The problem of this solution was too much code repetition. Also, I used 2 decorators instead of 1. Gilch said one way to do this was to decorate a class instead of a function.

Decorating a Class

Gilch gave me the test case, and asked me to write the decorator:

@if_(1>2)
class Result:
    def then():
        print("Yes.")
        return 42
    def else_():
        print("No.")
        return -1

Eventually I came up with the decorator:

def if_(condition):
    def decorator(c):
        if condition:
            return c.then()
        else:
            return c.else_()
    return decorator
$ python3.9 -i decorating_a_class.py 
Yes.
>>> Result
42

with condition reversed (@if_(1<2)):

$ python3.9 -i decorating_a_class.py 
No.
>>> Result
-1

Decorating a Class, Example 2

Gilch gave me another test case for which I needed to write the decorator:

@sign(x)
class Result:
    def positive():
        print('pos')
    def zero():
        print('zero')
    def negative():
        print('neg')

I modified the solution from the previous question to answer this one:

def sign(x):
    def decorator(c):
        if x > 0:
            return c.positive()
        elif x == 0:
            return c.zero()
        else:
            return c.negative()
    return decorator

This was basically the same example as the previous one in the sense that I didn't need to change the structure of the decorator, only the specific functions within the innermost layer.

Making Branching Statements without Using if

I asked gilch if there were ways to make if statement without using if. Gilch gave me some examples:

  • return [c.then,c.else][not condition]()
  • return {True:c.then,False:c.else_}[bool(condition)]()

A More Difficult Example

Gilch gave me another test case:

@if_(1<2)
def then():
    print('yes')
    return 42
@then
def result():
    print('no')
    return -1
assert result == 42

And when the first line is changed to @if_(1>2), assert result == -1 should pass.

I figured out how to make the first branch work but not the second. At this point, it was getting quite late in my timezone and by brain was slowing down after hours of coding, so I told gilch I'd work on it the next day.

When I picked this problem up the next day, all of the wobbly parts of my understanding came back to bite me. I was running into a bunch of error messages like TypeError: 'NoneType' object is not callable and TypeError: <function> missing 1 required positional argument: 'greetings' (or TypeError: <function> takes 1 positional argument but 2 were given), and since the error messages would point at the line where result was defined instead of the line within the decorator where the problem was, I was struggling a lot debugging my code. I poked around the internet a lot to look at other examples as an attempt to understand how to expand decorators in my head, and eventually came across this slide from a PyCon talk that made things a bit more clear to me.

I sent gilch my solution:

def if_(condition):
    def decorator(function):
        if condition:
            def wrapper(*x):
                return function()
            return wrapper
        else:
            def decorator2(function):
                return function()
            return decorator2
    return decorator

# original test cases below
@if_(1<2)
def then():
    print('yes')
    return 42
@then
def result():
    print('no')
    return -1
assert result == 42
$ python3.9 -i cases.py
yes
>>> result
42

With the condition flipped:

$ python3.9 -i cases.py
no
>>> result
-1

Gilch noticed my confusion: "The way you named things indicates a lack of understanding. Naming should have been more like this."

def if_(condition):
    def decorator(function):
        if condition:
            def decorator2(f):
                return function()
            return decorator2
        else:
            def decorator2(f):
                return f()
            return decorator2
    return decorator

Alternatively, gilch provided two more solutions:

def if_(condition):
    def decorator(function):
        if condition:
            return lambda _: function()
        else:
            return lambda f: f()
    return decorator
def if_(condition):
    def decorator(then_func):
        def decorator2(else_func):
            return (then_func if condition else else_func)()
        return decorator2
    return decorator

"They're both a decorator you apply to the second definition, regardless of which branch is taken. So calling the first one a wrapper is the wrong way to think of it," explained gilch.

Observations

  1. The concept of decorator being syntactic sugar of calling the decorator on a function then reassigning the result to the variable that used to be the function somehow didn't stick for me. When gilch asked me to write the desugared version of functions, I kept thinking about further expanding the functions.
  2. There are a few things that I was not used to thinking about: functions of functions, functions that use variables that are in scope but don't get passed in as parameters, functions that return callables. All these made decorators--especially parameterized decorators--"hard to fit into my head at once," using gilch's remark.
  3. Gilch's teaching was primarily puzzle-based, which is quite different from the lecture-based learning that I have done. This is pretty cool, because for most of my life, solving problems has been in the context of some evaluation. Stress sucked the fun out of puzzles for me when I had trouble decoupling my self worth and the outcomes of those evaluations. Now I'm solving puzzles for no other reason beyond learning, I've realized this is also an opportunity to unlearn.

29

18 comments, sorted by Highlighting new comments since Today at 6:20 PM
New Comment

(using a lambda as a decorator requires Python 3.9 or later, for anyone wondering what's going on here)

For future readers wondering what the fuss is about, in older versions of Python, the decorator syntax was restricted to identifier expressions possibly containing attribute access . and up to one call () at the end.

Of course, the call could always contain arbitrary expressions as arguments, which made it easy to sneak in arbitrary decorator expressions via an identity function.

def me(x):
    return x

@me(lambda f: print("hi"))
def add(a, b):
    return a + b

This was the topic of one of my StackOverflow answers.

Meta: This seems like a good format to spread knowledge from people who have it and are willing to teach, but either suck at writing or don't want to spend time writing and editing. Become an apprentice in return for writing a guide.

Is this still worth doing? Is the community getting anything out of these? The first post in this series has 39 voters so far. This one only has seven.

Oh, I forgot that karma 2.0 is not the number of voters.

Dunno, maybe diminshing returns, or people are on holidays? The drop from 39 to currently 9 seems significant, but it's still just two data points. If you don't mind writing, I would suggest to post another part in September and see whether the reaction increases.

From my perspective (although I have upvoted both, so I am not the right audience for your question) there are diminishing returns. I used Python briefly in the past, and maybe will use it in future again, but I am not using it currently... so I was kinda excited to see some cool tricks about something I studied for a while, but more cool tricks are at this moment only insight porn for me. On the other hand, if I ever use Python in future again, I wish I would remember to read these articles again.

konstell seems to be getting all the Karma though. Is there a way to do dual authorship on a post or something? Not sure what to do about it.

At the moment, users can message LW admins to add more authors, so I asked to add you as an author too. However, "only the first author gets karma, regrettably."

Feedback

In the "Preprint" example, you use @wraps without explaining it. I think it's worth noting it is imported from functools and it's general purpose?

The section "Branching without if" is a bit confusing, because it is unclear if those examples work or need more code.

Discussion

This might be out of scope of the mentorship, but I'd like gilch's opinion/heuristics on:

  1. Good decorator use vs. abuse
  2. The argument "Python would not need decorators if it supported multi-line anonymous functions"

Re #1.

Readability counts.
[...]
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
—excerpts from The Zen of Python

This part of Python's culture was a reaction to Perl's motto, "There's more than one way to do it". Perl has been derided as a Write-only language, and despite its initial popularity, has been far eclipsed by Python. We can speculate about the various reasons, but I think this is one of them. Other ecosystems have their own cultures which emphasize different tradeoffs. Ruby, for example, seems to have inherited Perl's take.

The Zen is full of allusions and apparent contradictions. It's not meant to be obeyed as much as meditated upon; less about answers, and more about which questions to ask in the first place. So take my commentary as interpretation, not law.

Among other things, this part of The Zen is pointing out that code is read more often than it is written. It's more important to make code easy to read than easy to write. It's possible to write bad code in any language, but "bad" is partly a cultural interpretation. (And partly not.)

In Python, in the interest of readability, one should first do things in the most obvious way, until one has very good reason to do otherwise.

But "obvious" is also cultural. What is idiomatic in C is not, in Python, idiomatic (or "pythonic"), the "Dutch" part alluding to Guido van Rossum, the creator of Python, who made a lot of judgement calls about how Python was going to be, and what counts as "normal".

So "obvious" here means "obvious to acculturated Python programmers". It's a warning about being "too clever", and thereby being obtuse. But clever is OK when it's easy to read! Are you trying to show off or are you trying to write good code? If you are an acculturated Python programmer, then you can judge what's obvious and isn't. But culture isn't a static thing. If you're a part of it, and you know better, you can push boundaries a little.

(To read all of The Zen, import this.)

My exercises for konstell are not necessarily pythonic. They illustrate particular points I was trying to teach; weaknesses I noticed in her understanding at the time. That you can do something doesn't mean you should.

So, like anything else in Python, my main heuristic would be, does your decorator make the code easier or harder to read, for an acculturated Python programmer? Does it have benefits worth the costs? Decorators are a Python feature, and they're used a lot. Excessive length is bad for readability too, and often neglected. Being more concise can be worth being slightly less obvious. Does it make tooling easier or harder to use? Does it make testing more or less difficult?

Re #2.

The multi-line lambda one always makes me want to facepalm. I hear it a lot, but this one is so confused that it's not even wrong.

First, lambdas in Python can have as many lines as you want. That makes the premise invalid. Seriously. My Hissp project is a Lisp compiler that targets a subset of Python. Its lambdas have the implicit PROGN typical of Lisp, and they compile to Python lambdas just fine. You could wrap your entire Hissp module in a progn and it will compile to a lambda expression that goes on for pages.

So why are people confused about this? I think they're conflating "lines of code" with "statements", which are not at all the same thing. It's true that certain kinds of Python statements typically fit on one line, but none of them have to, and many kinds (block statements, e.g. try-except/try-finally) typically don't.

So let's try to steelman this: even multi-line lambdas in Python can't directly contain statements. (Or, from a certain point of view, they contain only one: an implicit return. But they can call exec() on a string that does, or call other things that do.)

Second, true functional languages don't have statements to begin with, only expressions. (Or, from a certain point of view, they only have "expression statements".) Statements are a holdover from assembly, when the original Fortran mixed math expressions with machine code instructions (the "statements").

When programming in the functional style, which is when you want lambdas, you don't use statements anyway. Expressions are all you need! You don't even need a progn unless you have side effects, which is also not functional style.

So then the argument becomes "Python would not need decorators if anonymous functions could have statements."

Now what does the "need" part mean? Decorators are just syntactic sugar. You can get exactly the same behavior without them, so what use are decorators at all? Let's look at what the sugar does:

def <name>(<args>):
    <body>

<name> = (<decorator expression>)(<name>)

becomes

@<decorator expression>
def <name>(<args>):
    <body>

"Need" might be overstated. But why is the decorator better? The main reason is that it eliminates the duplication of <name>, which appears three times in the desugared version. It's also shorter and allows you to write the decorator expression before the function. Do lambdas have <name>s? No.

So what are they suggesting we could do instead? It's probably

<name> = (<decorator expression>)(
    lambda <args>:
        <body>
)

which does already work in Python if <body> doesn't have statements (and it wouldn't in the functional style). But we're still missing the function's docstring, and its __name__ attribute will be '(lambda)' instead of <name>. Hypothetically, to fix these, it then becomes something like,

<name> = (<decorator expression>)(
    def <name>(<args>):
        <doc>
        <body>
)

This doesn't work because def isn't an expression in Python. It's not about lambdas anymore.

Now the steelman argument has become "Python would not want decorators if function definitions were expressions."

But we can see that the decorator version is still better. It doesn't duplicate the <name>. It doesn't need another layer of indentation, which gets even worse when nesting these. Can we fix this? Maybe?

def defn(*decorators):
    def _(name, f):
        while decorators:
            f = decorators.pop()(f)
        globals()[name] = f
        f.__name__ = name  # etc.
    return _

defn(<decorator expression>)(
    "<name>", lambda <args>:
        <body>
)

This only assigns to the top level because it uses globals, even if it's nested in a class or function. Perhaps if we had some kind of preprocessor macro that expanded to an assignment? But now we've just re-implemented decorators.

The section "Branching without if" is a bit confusing, because it is unclear if those examples work or need more code.

The relevant context is the earlier definition of @if_.

def if_(condition):
    def decorator(c):
        if condition:
            return c.then()
        else:
            return c.else_()
    return decorator

So

def if_(condition):
    def decorator(c):
        return [c.then, c.else][not condition]()
    return decorator

would have the same behavior, and does not itself have an if statement. I've implemented if without using if.

Ah, I missed a section on @wraps. Added it here.

Also renamed "Branching without if" to "Making Branching Statements without Using if"

Also added some command line outputs for a couple examples at the end.

functions that use variables that are in scope but don't get passed in as parameters

A function object combining both executable code and some variables from the surrounding lexical environment is called a lexical closure.

Different languages handle this kind of thing differently. Closures were motivated by lambda calculus, so they're a common feature of languages with higher-order functions. (Although lambda calculus itself has no such concept, it emerges naturally as an optimization when trying to write an efficient evaluator for it.)

Python added these with PEP 227 when it gained lexical scoping, years before I started using the language.

I asked gilch if it was possible to pipe outputs from python shell to bash tools, gilch answered that we could not do it directly

Maybe not entirely accurate as worded. I don't remember exactly how this part of the conversation went, but (for example)

>>> import os; from unittest.mock import patch
>>> with os.popen('less', 'w') as p, patch('sys.stdout.write', p.write):
...     copyright
... 

would pop up less with the builtin copyright statement. Not as simple as the | in Bash. It might take me a while to explain how exactly how this works, but it's basically monkeypatching stdout to pipe to less. Redirecting outputs of the python shell is not the normal way of doing things. You can be more direct about it.

You can use bash to pipe things into or out of calls to Python (or both). Python can also use variations on popen to pipe to or from other utilities, although it's not nearly as easy as |.

I can't reproduce the help(bob). Where is the # @wraps(function) coming from?

>>> def preprint(greetings): # factory
...     def greet_by_name(function): # decorator
...         def wrapper(): # replaces the function being decorated
...             print(greetings)
...             return function()
...         return wrapper
...     return greet_by_name
...
>>> @preprint("hi")
... def bob():
...     """
...     prints 'bob'
...     """
...     print("bob")
...
>>> help(bob)
Help on function wrapper in module __main__:

wrapper()

Sorry, this is what happens when I don't keep a good trace of changes and try to reconstruct code snippets. When I was writing this post, the version of the function in my file was

 def preprint(greetings): # factory
    def greet_by_name(function): # decorator
        @wraps(function)
        def wrapper(): # replaces the function being decorated
            print(greetings)
            return function()
        return wrapper
    return greet_by_name

In order to reproduce the effect on docstring of bob without using wraps(function), I simply commented that line out, so the function definition became

def preprint(greetings): # factory
    def greet_by_name(function): # decorator
        # @wraps(function)
        def wrapper(): # replaces the function being decorated
            print(greetings)
            return function()
        return wrapper
    return greet_by_name

And # @wraps(function) became the docstring...

I have fixed this in the post.

Looks like help() will fall back to reading a comment above a definition if it can't find a docstring, but only if it's in an accessible source file. Functions do know their line numbers for debugging purposes. If you try this with a definition in the REPL, or in an exec'd string or .pyc file (without corresponding source), this won't work, because there is no file to read the comments from, unlike a docstring which would be available at runtime when read from any of these sources.

Also, if you modify the source file after importing the object, then the line numbers won't match up and it might find the wrong comment (when it reads the modified source file to look for comments). This can't happen with docstrings either, since they're saved when the object is created.

See also inspect.getcomments().

Gilch illustrated the difference between a function and a callable

This part still seems confused. What you remember about this part isn't quite what I was trying to say. Functions are callables, but they're not the only type of callable. Classes are also callable, for example. Lambdas are of the function type in Python, so these aren't exactly separate categories.

>>> type(lambda:0)
<class 'function'>

Callables are anything you can call; they understand the function call operator. You can use the builtin predicate callable to tell if an object is callable or not.