Thank you for sharing!
This is a win/win move, because you teach other people interested in Python, thus multiplying the effect of gilch's tutoring, and at the same time probably improve your own understanding and remembering. (In my experience, when I try to teach something to others, my own blind spots and memorized passwords become visible.)
This reminded me of lvalue and rvalue in C++, so I asked if they were the same thing as store vs. load context. They were.
This one might need more clarification. I said they were the same idea, not the same thing. In a pointer-based language like C, a pointer is a kind of first-class integer value. You can pass them into and out of functions and you can use one as a store target in an assignment. You can even do arithmetic with them. Pointers are memory addresses.
However, in a memory-safe garbage-collected language like Python, references and values are categorically different things, so it's incorrect to call store targets "values" at all. References are, of course, implemented using pointers behind the scenes, but references are not first-class values like a pointer would be. You can only store to locations, and you can't directly pass around the location itself like a pointer value, although you can pass around container objects which have assignable locations. In Python, this usually means a dict (or an object backed by one), but there are a few other species, like lists. Dict keys and list indexes are values, but they aren't locations by themselves. To be assignable, you have to combine them with their container.
It's done this way so that an object's memory can be instantly freed when the object has no more references. (And eventually freed when it's no longer reachable. These can happen at different times when there are reference cycles.)
I verbalized the code in my head out loud, then asked how we'd convert the types of the function return value to string before appending ", meow" to it. Gilch suggested f"{foo(x, y)}, meow" and we had our third decorator.
We then applied decorators in different orders to show that multiple decorators were allowed, and that the order of decorators decided the order of application.
That's not the order I remember. I recall you tried foo(x, y) + ", meow" first, as I expected from how I worded the task. You tried applying the decorators in a different order for add than you did for subtract, I think because you expected one of them to be a type error, but weren't sure which. That the order of the decorators can matter and what order they apply were points I was trying to illustrate with this task.
After that, I pointed out the f-string would work even if foo doesn't return a string, and that would make the @convert_to_str redundant, instead of necessary as an adapter.
Thanks for pointing this out, you're right. Even when I have your half of the transcript available to me, I still found it sometimes hard to recall what exactly I tried when I was confused about a concept.
A couple weeks ago Zvi made an apprentice thread. I have always wanted to be someone's apprentice, but it didn't occur to me that I could just ...ask to do that. Mainly I was concerned about this being too big of an ask. I saw gilch's comment offering to mentor Python programming. I want to level up my Python skills, so I took gilch up on the offer. In a separate comment, gilch posed some questions about what mentors and/or the community get in return. I proposed that I, as the mentee, document what I have learned and share it publicly.
Yesterday we had our first session.
Background
I had identified that I wanted to fill gaps in my Python knowledge, two of which being package management and decorators.
Map and Territory
Gilch started by saying that "even senior developers typically have noticeable gaps," but building an accurate map of the territory of programming would enable one to ask the right questions. They then listed three things to help with that:
Documentation on the Python standard library. "You should at least know what's in there, even if you don't know how to use all of it. Skimming the documentation is probably the fastest way to learn that. You should know what all the operators and builtin functions do."
Structure and Interpretation of Computer Programs for computer science foundation. There are some variants of the book in Python, if one does not want to use Scheme.
CODE as "more of a pop book" on the backgrounds.
In my case, I minored in CS, but did not take operating systems or compilers. I currently work as a junior Python developer, so reading the Python standard library seems to be the lowest hanging fruit here, with SICP on the side, CODE on the back burner.
Decorators
The rest of the conversation consisted of gilch teaching me about decorators.
Gilch: Decorators are syntactic sugar.
@foo
def bar():
...means the same thing as
def bar():
...
bar = foo(bar)Decorators also work on classes.
@foo
class Bar:
...is the same as
class Bar:
...
Bar = foo(Bar)An Example from pytest
At this point I asked if decorators were more than that. I had seen decorators in pytest:
@pytest.fixture
def foo():
...
def test_bar(foo): # foo automatically gets evaluated inside the test
...Does this mean that, when foo is passed in test_bar as a variable, what gets passed in is actually something like pytest.fixture(foo)?
Gilch identified that there might be more than decorators involved in this example, so we left this for later and went back to decorators.
Decorators, Example 1
I started sharing my screen, gilch gave me the first instruction: Try making a decorator.
def test_decorator(foo):
return 42
@test_decorator
def bar():
print('hi')
print(bar())Then, before I ran the program, gilch asked me what I expected to happen when I run this program, to which I answered that hi and 42 would be printed to console. At this point, gilch reminded me that decorators were sugar, and asked me to write out the un-sugared translation of the function above. I wrote:
def bar():
bar = test_decorator(bar)
return barI ran the program, and was surprised by the error TypeError: 'int' object is not callable. I expected bar to still be a function, not an integer.
Gilch asked me to correct my translation of my program based on the result I saw. It took me a few more tries, and eventually they showed me the correct translation:
def bar():
print('hi')
bar = test_decorator(bar)Then I realized why I was confused--I had the idea that decorators modify the function they decorate directly (in the sense of modifying function definitions), when in fact the actions happen outside of function definitions.
Gilch explained: A decorator could [modify the function], but this one doesn't. It ignores the function and returns something else. Which it then gives the function's old name.
Decorators, Example 2
Gilch: Can you make a function that subtracts two numbers?
Me:
def subtract(a, b):
return a - bGilch: Now make a decorator that swaps the order of the arguments.
My first thought was to ask if there was any way for us to access function parameters the same way we use sys.argv to access command line arguments. But gilch steered me away from that path by pointing out that decorators could return anything. I was stuck, so gilch suggested that I try return lambda x, y: y-x.
Definition Time
My program looked like this at this point:
@swap_order
def subtract(a, b):
return a - b
def swap_order(foo):
return lambda x, y: y - xPyCharm gave me an warning about referencing swap_order before defining it. Gilch explained that decoration happened at definition time, which made sense considering the un-sugared version.
Interactive Python
Up until this point, I had been running my programs with the command python3 <file>. Gilch suggested that I run python3 -i <file> to make it interactive, which made it easier to experiment with things.
Decorators, Example 2
Gilch: Now try an add function. Decorate it too.
Me:
def swap_order(foo):
return lambda x, y: y - x
@swap_order
def subtract(a, b):
return a - b
@swap_order
def add(a, b):
return a + bGilch then asked, "What do you expect the add function to do after decoration?" To which I answered that the add function would return the value of its second argument subtracted by the first argument. The next question gilch asked was, "Can you modify the decorator to swap the arguments for both functions?"
I started to think about sys.argv again, then gilch hinted, "You have 'foo' as an argument." I then realized that I could rewrite the return value of the lambda function:
def swap_order(foo):
return lambda x, y: foo(y, x)I remarked that we'd see the same result from add with or without the decorator. Gilch asked, "Is addition commutative in Python?" and I immediately responded yes, then I realized that + is an overloaded operator that would work on strings too, and in that case it would not be commutative. We tried with string inputs, and indeed the resulting value was the reverse-ordered arguments concatenated together.
Gilch: Now can you write a decorator that converts its result to a string?
I wrote:
def convert_to_str(foo):
return str(foo())It was not right. I then tried
def convert_to_str(foo):
return strand it was still not right. Finally I got it:
def convert_to_str(foo):
lambda x, y: str(foo(x, y))There was some pair debugging that gilch and I did before I reached the answer. Looking at the mistakes I've made here, I see that I still hadn't grasped the idea that decorators would return functions that transform the results of other functions, not the transformed result itself.
Gilch: Try adding a decorator that appends ", meow." to the result of the function.
I verbalized the code in my head out loud, then asked how we'd convert the types of the function return value to string before appending ", meow" to it. Gilch suggested f"{foo(x, y)}, meow" and we had our third decorator.
We then applied decorators in different orders to show that multiple decorators were allowed, and that the order of decorators decided the order of application.
Splat
When we were writing the convert_to_str decorator, I commented that this would only work for functions that take in exactly 2 arguments. So gilch asked me if I was familiar with the term "unpacking" or "splat." I knew it was something like ** but didn't have more knowledge than that.
How Many Arguments
Gilch asked me, "How many arguments can print() take?" To which I answered "infinite." They then pointed out that it was different from infinite--zero would be valid, or one, or two, and so on. So the answer is "any number, " and the next challenge would be to make convert_to_str work with any number of arguments.
print()
We tried passing different numbers of arguments into print(), and sure enough it took any number of arguments. Here, gilch pointed out that print actually printed out a newline character by default, and the default separator was a space. They also pointed out that I could use the help(print) command to access the doc in the terminal without switching to my browser.
type(_)
Gilch pointed out that I could use the command type(_) to get the type of the previous value in the console, without having to copy and paste.
Splat
To illustrate how splat worked, gilch gave me a few commands to try. I'd say out loud what I expected the result to be before I ran the code. Sometimes I got what I expected; sometimes I was surprised by the result, and gilch would point out what I had missed. To illustrate splat in arrays, gilch gave two examples: print(1,2,3,*"spam", sep="~") and print(1,2,*"eggs",3,*"spam", sep="~"). Then they showed me how to use ** to construct a mapping: (lambda **kv: kv)(foo=1, bar=2)
Dictionary vs. Mapping
We went off on a small tangent on dictionary vs. mapping because gilch pointed out that dictionary was not the only type of mapping and tuple is no the only type of iterable. I asked if there were other types of mapping in Python, and they listed OrderedDict as a subtype and the Mapping abstract class.
Parameter vs. Argument, Packing vs. Unpacking
At this point gilch noticed that I kept using the word "unpacking." I also noticed that I was using the term "argument" and "parameter" interchangeably here. Turns out the distinction is important here--the splat operator used on a parameter packs values in a tuple; used on an argument unpacks iterable into separate values. For example, in (lambda a, b, *cs: [a, b, cs])(1,2,3,4,5), cs is a parameter and * packs the values 3, 4, 5 into a tuple; in print(*"spam", sep="~"), "spam" is an argument and * unpacks it into individual characters.
Dictionaries
Gilch gave me another example: Try {'x':1, **dict(foo=2,bar=3), 'y':4}. I answered that it would return a dictionary with four key-value pairs, with foo and bar also becoming keys. Gilch then asked, "in what order?" To which I answered "dictionaries are not ordered."
"Not true anymore," gilch pointed out, "Since Python 3.7, they're guaranteed to remember their insertion order." We looked up the Python documentation and it was indeed the case. We tried dict(foo=2, **{'x':1,'y':4}, bar=3) and got a dictionary in a different order.
Hashable Types
I asked if there was any difference in defining a dictionary using {} versus dict(). Gilch compared two examples: {42:'spam'} works and dict(42='spam') doesn't. They commented that keys could be any hashable type, but keyword arguments were always keyed by identifier strings. The builtin hash() only worked on hashable types.
I don't fully understand the connection between hashable types and identifier strings here, it's something that I'll clarify later.
Parameter vs. Argument, Packing vs. Unpacking
Gilch gave another example: a, b, *cs, z = "spameggs"
I made a guess that cs would be an argument here, so * would be unpacking, but then got stuck on what cs might be. I tried to run it:
>>> a, b, *cs, z = "spameggs"
>>> a
's'
>>> b
'p'
>>> cs
['a', 'm', 'e', 'g', 'g']
>>> z
's'Gilch pointed out that cs was a store context, not a load context, which made it more like a parameter rather than an argument. Then I asked what store vs. load context was.
Context
Gilch suggested, import ast then def dump(code): return ast.dump(ast.parse(code)). Then something like dump("a = a") would return a nexted object, in which we can locate the ctx value for each variable.
This reminded me of lvalue and rvalue in C++, so I asked if they were the same thing as store vs. load context. They were.
Splat
Gilch tied it all together, "So for a decorator to pass along all args and kwargs, you do something like lambda *args, **kwargs: foo(*args, **kwargs). Then it works regardless of their number. Arguments and keyword arguments in a tuple and dict by keyword. So you can add, remove, and reorder arguments by using decorators to wrap functions. You can also process return values. You can also return something completely different. But wrapping a function in another function is a very common use of decorators. You can also have definition-time side effects. When you first load the module, it runs all the definitions--This is still runtime in Python, but you define a function at a different time than when you call it. The decoration happens on definition, not on call."
We wrapped up our call at this point.
Observations