Roberto Alsina: Learning Serverless in GCP(9 hours, 8 minutes ago)

Usually, when I want to learn how to use a tool, the thing that works best for me is to try to build something using it. Watching someone build something instead is the second best thing.

So, join me while I build a little thing using "serverless" Google Cloud Platform, Python and some other bits and pieces.


Caveat: this was originally a twitter thread, so there will be typos and things. Sorry! Also it's possible that it will look better here in threaderapp


A thread by Roberto Alsina

Ready for an afternoon learning with uncle Roberto? Come along, today's topic is "serverless"! Start thread.

As usual, serverless seems a bit daunting. So, our code will run in that vast, anonymous, vaguely lovecraftian infrastructure? Like AWS or GCP?

Well, yes. But you only need be scared of Cthulhu if you have not seen Cthulhu as a puppy.

So, let's start with a puppy-like thing. Yesterday @quenerapu mentioned https://carbon.now.sh which shows pretty code snippets but just as images. Pretty? Good.

Only pretty and no copy/paste? Bad baby Cthulhu

So, let's do something about it. This is a script that will produce either a colorized image of code, or HTML. Yes, that's a screenshot. Yes, I notice the irony.

It produces, when ran as shown, this:

Is it pretty? No. But remember, no successful complicated thing grew out of a complicated thing. As long as the inputs and outputs of our initial code are within the same postal code as what we really would like to have, it's good enough for now. So, what can I do with that code? I can do a flask app.

And if I run it, it works

So, this is the seed of an app that could replace https://carbon.now.sh ... it would need frontend, and (to make it better) something a lot like a URL shortener, but that's besides the point.

And now I can choose different paths to follow.

One way I could go forward is to deploy it to my server. Because I have a server. It's a tiny server, and it's very cheap, but this? It can run it a million times a day and I won't notice. But I said this thread is about serverless, right? So you know I am not doing that.

There are 2 main serverless environments I could try (because they are the 2 ones I know about)

* AWS Lambda
* Google cloud functions

Let's try google's. Later maybe we'll try the other one. But the important bit here is: doing serverless and doing "serverfull" is not really all that different. As long as:

1. Your endpoints are stateless
2. You don't rely on the filesystem for state
3. All the state you want, you put on the database

Now I have to go setup my billing information in Google Cloud, BRB. Done, there is a free tier and some free credit for starting up, and it's not going to cost any real money anyway.

So

I changed like, two lines.

And added requirements

It took some 30 seconds to deploy.

And it works for HTML output (…ering-force-268517.cloudfunctions.net/color-coding?f…)

But not for PNG, because while the code runs just fine the actual environment it's running on is ... limited.

So it has no fonts, which it needs to do the image.

These are the packages GCP provides in the image where our functions run. So, it's not the same as my desktop machine, to say the least.

Looks like we have liberation fonts, so "Liberation Mono" should be available? Yep. (…ering-force-268517.cloudfunctions.net/color-coding?l…) So, how do we make this useful?

Let's define a user story. I, a developer, want to show my code in twitter, but pretty and longer than 240 characters.

Also, I want people to be able to copy all or part of that text without typing it manually as if it were 1989. Now I will take a short break. Go have some coffee.

This uses this twitter feature: (developer.twitter.com/en/docs/tweets…)

So, if I make a second serverless function that generates something like that, I am almost there. Then it's a matter of UI. Mind you, it's perfectly ok to implement this as a flask app and then just once it does what I want it to do redeploy to google cloud.

The code looks exactly the same anyway.

That is just a matter of some basic string templating.

So, there you go, that Google cloud function is all the backend you need to implement something like https://carbon.now.sh

But I don't really want to, so you go ahead.

EOT

(Tomorrow: running this on AWS using Zappa and on GCP using gcloud)

Mike Driscoll: Python 101 2nd Edition Kickstarter is Live!(14 hours, 27 minutes ago)

I am excited to announce that my newest book, Python 101, 2nd Edition is launching on Kickstarter today!

Python 101 2nd Ed KickstarterClick the Photo to Jump to Kickstarter

Python 101 holds a special place in my heart as it was the very first book I ever wrote. Frankly, I don’t think I would have even written a book if it weren’t for the readers of this blog who encouraged me to do so.

The new edition of Python 101 will be an entirely new, rewritten from scratch, book. While I will be covering most of the same things as in the original, I have reorganized the book a lot and I am adding all new content. I have also removed old content that is no longer relevant.

I hope you will join me by backing the book and giving me feedback as I write it so that I can put together a really great learning resource for you!

The post Python 101 2nd Edition Kickstarter is Live! appeared first on The Mouse Vs. The Python.

Real Python: A Guide to the Newer Python String Format Techniques(14 hours, 29 minutes ago)

In the previous tutorial in this introductory series, you learned how to format string data using the string modulo operator. The string modulo operator is useful, and it’s good for you to be familiar with it because you’re likely to encounter it in older Python code. However, there are two newer ways that you can use Python to format strings that are arguably more preferable.

In this tutorial, you’ll learn about:

  1. The string .format() method
  2. The formatted string literal, or f-string

You’ll learn about these formatting techniques in detail and add them to your Python string formatting toolkit. Note that there’s a standard module called string containing a class called Template, which provides some string formatting through interpolation. The string modulo operator provides more or less the same functionality, so you won’t cover string.Template here.

Free Bonus: Click here to get access to a chapter from Python Tricks: The Book that shows you Python's best practices with simple examples you can apply instantly to write more beautiful + Pythonic code.

The Python String .format() Method

The Python string .format() method was introduced in version 2.6. It’s similar in many ways to the string modulo operator, but .format() goes well beyond in versatility. The general form of a Python .format() call is shown below:

<template>.format(<positional_argument(s)>, <keyword_argument(s)>)

Note that this is a method, not an operator. You call the method on <template>, which is a string containing replacement fields. The <positional_arguments> and <keyword_arguments> to the method specify values that are inserted into <template> in place of the replacement fields. The resulting formatted string is the method’s return value.

In the <template> string, replacement fields are enclosed in curly braces ({}). Anything not contained in curly braces is literal text that’s copied directly from the template to the output. If you need to include a literal curly bracket character, like { or }, in the template string, then you can escape this character by doubling it:

>>>
>>> '{{ {0} }}'.format('foo')
'{ foo }'

Now the curly braces are included in your output.

The String .format() Method: Arguments

Let’s start with a quick example to get you acquainted before you dive into more detail on how to use this method in Python to format strings. For review, here’s the first example from the previous tutorial on the string modulo operator:

>>>
>>> print('%d %s cost $%.2f' % (6, 'bananas', 1.74))
6 bananas cost $1.74

Here, you used the string modulo operator in Python to format the string. Now, you can use Python’s string .format() method to obtain the same result, like this:

>>>
>>> print('{0} {1} cost ${2}'.format(6, 'bananas', 1.74))
6 bananas cost $1.74

In this example, <template> is the string '{0} {1} cost ${2}'. The replacement fields are {0}, {1}, and {2}, which contain numbers that correspond to the zero-based positional arguments 6, 'bananas', and 1.74. Each positional argument is inserted into the template in place of its corresponding replacement field:

Python string format, positional parametersUsing The String .format() Method in Python to Format a String With Positional Arguments

The next example uses keyword arguments instead of positional parameters to produce the same result:

>>>
>>> print('{quantity} {item} cost ${price}'.format(
...     quantity=6,
...     item='bananas',
...     price=1.74))
6 bananas cost $1.74

In this case, the replacement fields are {quantity}, {item}, and {price}. These fields specify keywords that correspond to the keyword arguments quantity=6, item='bananas', and price=1.74. Each keyword value is inserted into the template in place of its corresponding replacement field:

Python string format and keyword parametersUsing The String .format() Method in Python to Format a String With Keyword Arguments

You’ll learn more about positional and keywords arguments there in the next tutorial in this introductory series, which explores functions and argument passing. For now, the two sections that follow will show you how these are used with the Python .format() method.

Positional Arguments

Positional arguments are inserted into the template in place of numbered replacement fields. Like list indexing, the numbering of replacement fields is zero-based. The first positional argument is numbered 0, the second is numbered 1, and so on:

>>>
>>> '{0}/{1}/{2}'.format('foo', 'bar', 'baz')
'foo/bar/baz'

Note that replacement fields don’t have to appear in the template in numerical order. They can be specified in any order, and they can appear more than once:

>>>
>>> '{2}.{1}.{0}/{0}{0}.{1}{1}.{2}{2}'.format('foo', 'bar', 'baz')
'baz.bar.foo/foofoo.barbar.bazbaz'

When you specify a replacement field number that’s out of range, you’ll get an error. In the following example, the positional arguments are numbered 0, 1, and 2, but you specify {3} in the template:

>>>
>>> '{3}'.format('foo', 'bar', 'baz')
Traceback (most recent call last):
  File "<pyshell#26>", line 1, in <module>
    '{3}'.format('foo', 'bar', 'baz')
IndexError: tuple index out of range

This raises an IndexError exception.

Starting with Python 3.1, you can omit the numbers in the replacement fields, in which case the interpreter assumes sequential order. This is referred to as automatic field numbering:

>>>
>>> '{}/{}/{}'.format('foo', 'bar', 'baz')
'foo/bar/baz'

When you specify automatic field numbering, you must provide at least as many arguments as there are replacement fields:

>>>
>>> '{}{}{}{}'.format('foo','bar','baz')
Traceback (most recent call last):
  File "<pyshell#27>", line 1, in <module>
    '{}{}{}{}'.format('foo','bar','baz')
IndexError: tuple index out of range

In this case, there are four replacement fields in the template but only three arguments, so an IndexError exception occurs. On the other hand, it’s fine if the arguments outnumber the replacement fields. The excess arguments simply aren’t used:

>>>
>>> '{}{}'.format('foo', 'bar', 'baz')
'foobar'

Here, the argument 'baz' is ignored.

Note that you can’t intermingle these two techniques:

>>>
>>> '{1}{}{0}'.format('foo','bar','baz')
Traceback (most recent call last):
  File "<pyshell#28>", line 1, in <module>
    '{1}{}{0}'.format('foo','bar','baz')
ValueError: cannot switch from manual field specification to automatic field
 numbering

When you use Python to format strings with positional arguments, you must choose between either automatic or explicit replacement field numbering.

Keyword Arguments

Keyword arguments are inserted into the template string in place of keyword replacement fields with the same name:

>>>
>>> '{x}/{y}/{z}'.format(x='foo', y='bar', z='baz')
'foo/bar/baz'

In this example, the values of the keyword arguments x, y, and z take the place of the replacement fields {x}, {y}, and {z}, respectively.

If you refer to a keyword argument that’s missing, then you’ll see an error:

>>>
>>> '{x}/{y}/{w}'.format(x='foo', y='bar', z='baz')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'w'

Here, you specify replacement field {w}, but there’s no corresponding keyword argument named w. Python raises a KeyError exception.

While you have to specify positional arguments in sequential order, but you can specify keyword arguments in any arbitrary order:

>>>
>>> '{0}/{1}/{2}'.format('foo', 'bar', 'baz')
'foo/bar/baz'
>>> '{0}/{1}/{2}'.format('bar', 'baz', 'foo')
'bar/baz/foo'

>>> '{x}/{y}/{z}'.format(x='foo', y='bar', z='baz')
'foo/bar/baz'
>>> '{x}/{y}/{z}'.format(y='bar', z='baz', x='foo')
'foo/bar/baz'

You can specify both positional and keyword arguments in one Python .format() call. Just note that, if you do so, then all the positional arguments must appear before any of the keyword arguments:

>>>
>>> '{0}{x}{1}'.format('foo', 'bar', x='baz')
'foobazbar'
>>> '{0}{x}{1}'.format('foo', x='baz', 'bar')
  File "<stdin>", line 1
SyntaxError: positional argument follows keyword argument

In fact, the requirement that all positional arguments appear before any keyword arguments doesn’t apply only to Python format methods. This is generally true of any function or method call. You’ll learn more about this in the next tutorial in this series, which explores functions and function calls.

In all the examples shown so far, the values you passed to the Python .format() method have been literal values, but you may specify variables as well:

>>>
>>> x = 'foo'
>>> y = 'bar'
>>> z = 'baz'
>>> '{0}/{1}/{s}'.format(x, y, s=z)
'foo/bar/baz'

In this case, you pass the variables x and y as positional parameter values and z as a keyword parameter value.

The String .format() Method: Simple Replacement Fields

As you’ve seen, when you call Python’s .format() method, the <template> string contains replacement fields. These indicate where in the template to insert the arguments to the method. A replacement field consists of three components:

{[<name>][!<conversion>][:<format_spec>]}

These components are interpreted as follows:

Component Meaning
<name> Specifies the source of the value to be formatted
<conversion> Indicates which standard Python function to use to perform the conversion
<format_spec> Specifies more detail about how the value should be converted

Each component is optional and may be omitted. Let’s take a look at each component in more depth.

The <name> Component

The <name> component is the first portion of a replacement field:

{[<name>][!<conversion>][:<format_spec>]}

<name> indicates which argument from the argument list is inserted into the Python format string in the given location. It’s either a number for a positional argument or a keyword for a keyword argument. In the following example, the <name> components of the replacement fields are 0, 1, and baz, respectively:

>>>
>>> x, y, z = 1, 2, 3
>>> '{0}, {1}, {baz}'.format(x, y, baz=z)
'1, 2, 3'

If an argument is a list, then you can use indices with <name> to access the list’s elements:

>>>
>>> a = ['foo', 'bar', 'baz']
>>> '{0[0]}, {0[2]}'.format(a)
'foo, baz'
>>> '{my_list[0]}, {my_list[2]}'.format(my_list=a)
'foo, baz'

Similarly, you can use a key reference with <name> if the corresponding argument is a dictionary:

>>>
>>> d = {'key1': 'foo', 'key2': 'bar'}
>>> d['key1']
'foo'
>>> '{0[key1]}'.format(d)
'foo'
>>> d['key2']
'bar'
>>> '{my_dict[key2]}'.format(my_dict=d)
'bar'

You can also reference object attributes from within a replacement field. In the previous tutorial in this series, you learned that virtually every item of data in Python is an object. Objects may have attributes assigned to them that are accessed using dot notation:

obj.attr

Here, obj is an object with an attribute named attr. You use dot notation to access the object’s attribute. Let’s see an example. Complex numbers in Python have attributes named .real and .imag that represent the real and imaginary portions of the number. You can access these using dot notation as well:

>>>
>>> z = 3+5j
>>> type(z)
<class 'complex'>
>>> z.real
3.0
>>> z.imag
5.0

There are several upcoming tutorials in this series on object-oriented programming, in which you’ll learn a great deal more about object attributes like these.

The relevance of object attributes in this context is that you can specify them in a Python .format() replacement field:

>>>
>>> z
(3+5j)
>>> 'real = {0.real}, imag = {0.imag}'.format(z)
'real = 3.0, imag = 5.0'

As you can see, it’s relatively straightforward in Python to format components of complex objects using the .format() method.

The <conversion> Component

The <conversion> component is the middle portion of a replacement field:

{[<name>][!<conversion>][:<format_spec>]}

Python can format an object as a string using three different built-in functions:

  1. str()
  2. repr()
  3. ascii()

By default, the Python .format() method uses str(), but in some instances, you may want to force .format() to use one of the other two. You can do this with the <conversion> component of a replacement field. The possible values for <conversion> are shown in the table below:

Value Meaning
!s Convert with str()
!r Convert with repr()
!a Convert with ascii()

The following examples force Python to perform string conversion using str(), repr(), and ascii(), respectively:

>>>
>>> '{0!s}'.format(42)
'42'
>>> '{0!r}'.format(42)
'42'
>>> '{0!a}'.format(42)
'42'

In many cases, the result is the same regardless of which conversion function you use, as you can see in the example above. That being said, you won’t often need the <conversion> component, so you won’t spend a lot of time on it here. However, there are situations where it makes a difference, so it’s good to be aware that you have the capability to force a specific conversion function if you need to.

The <format_spec> Component

The <format_spec> component is the last portion of a replacement field:

{[<name>][!<conversion>][:<format_spec>]}

<format_spec> represents the guts of the Python .format() method’s functionality. It contains information that exerts fine control over how values are formatted prior to being inserted into the template string. The general form is this:

:[[<fill>]<align>][<sign>][#][0][<width>][<group>][.<prec>][<type>]

The ten subcomponents of <format_spec> are specified in the order shown. They control formatting as described in the table below:

Subcomponent Effect
: Separates the <format_spec> from the rest of the replacement field
<fill> Specifies how to pad values that don’t occupy the entire field width
<align> Specifies how to justify values that don’t occupy the entire field width
<sign> Controls whether a leading sign is included for numeric values
# Selects an alternate output form for certain presentation types
0 Causes values to be padded on the left with zeros instead of ASCII space characters
<width> Specifies the minimum width of the output
<group> Specifies a grouping character for numeric output
.<prec> Specifies the number of digits after the decimal point for floating-point presentation types, and the maximum output width for string presentations types
<type> Specifies the presentation type, which is the type of conversion performed on the corresponding argument

These functions are analogous to the components you’ll find in the string modulo operator’s conversion specifier, but with somewhat greater capability. You’ll see their capabilities explained more fully in the following sections.

The <type> Subcomponent

Let’s start with <type>, which is the final portion of <format_spec>. The <type> subcomponent specifies the presentation type, which is the type of conversion that’s performed on the corresponding value to produce the output. The possible values are shown below:

Value Presentation Type
b Binary integer
c Single character
d Decimal integer
e or E Exponential
f or F Floating point
g or G Floating point or Exponential
o Octal integer
s String
x or X Hexadecimal integer
% Percentage

These are like the conversion types used with the string modulo operator, and in many cases, they function the same. The following examples demonstrate the similarity:

>>>
>>> '%d' % 42
'42'
>>> '{:d}'.format(42)
'42'

>>> '%f' % 2.1
'2.100000'
>>> '{:f}'.format(2.1)
'2.100000'

>>> '%s' % 'foobar'
'foobar'
>>> '{:s}'.format('foobar')
'foobar'

>>> '%x' % 31
'1f'
>>> '{:x}'.format(31)
'1f'

However, there are some minor differences between some of the Python .format() presentation types and the string modulo operator conversion types:

Type .format() Method String Modulo Operator
b Designates binary integer conversion Not supported
i, u Not supported Designates integer conversion
c Designates character conversion, and the corresponding value must be an integer Designates character conversion, but the corresponding value may be either an integer or a single-character string
g, G Chooses between floating point or exponential output, but the rules governing the choice are slightly more complicated Chooses between floating point or exponential output, depending on the magnitude of the exponent and the value specified for <prec>
r, a Not supported (though the functionality is provided by the !r and !a conversion components in the replacement field) Designates conversion with repr() or ascii(), respectively
% Converts a numeric argument to a percentage Inserts a literal '%' character into the output

Next, you’ll see several examples illustrating these differences, as well as some of the added features of the Python .format() method presentation types. The first presentation type you’ll see is b, which designates binary integer conversion:

>>>
>>> '{:b}'.format(257)
'100000001'

The modulo operator doesn’t support binary conversion type at all:

>>>
>>> '%b' % 257
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: unsupported format character 'b' (0x62) at index 1

However, the modulo operator does allow decimal integer conversion with any of the d, i, and u types. Only the d presentation type indicates decimal integer conversion with the Python .format() method. The i and u presentation types aren’t supported and aren’t really necessary.

Next up is single character conversion. The modulo operator allows either an integer or single character value with the c conversion type:

>>>
>>> '%c' % 35
'#'
>>> '%c' % '#'
'#'

On the other hand, Python’s .format() method requires that the value corresponding to the c presentation type be an integer:

>>>
>>> '{:c}'.format(35)
'#'
>>> '{:c}'.format('#')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'c' for object of type 'str'

When you try to pass a value of a different type, you’ll get a ValueError.

For both the string modulo operator and Python’s .format() method, the g conversion type chooses either floating-point or exponential output, depending on the magnitude of the exponent and the value specified for <prec>:

>>>
>>> '{:g}'.format(3.14159)
'3.14159'

>>> '{:g}'.format(-123456789.8765)
'-1.23457e+08'

The exact rules governing the choice are slightly more complicated with .format() than they are with the modulo operator. Generally, you can trust that the choice will make sense.

G is identical to g except for when the output is exponential, in which case the 'E' will be in uppercase:

>>>
>>> '{:G}'.format(-123456789.8765)
'-1.23457E+08'

The result is the same as in the previous example, but this time with an uppercase 'E'.

Note: There are a couple of other situations where you’ll see a difference between the g and G presentations types.

Under some circumstances, a floating-point operation can result in a value that’s essentially infinite. The string representation of such a number in Python is 'inf'. It may also happen that a floating-point operation produces a value that can’t be represented as a number. Python represents this with the string 'NaN', which stands for Not a Number.

When you pass these values to Python’s .format() method, the g presentation type produces lowercase output, and G produces uppercase output:

>>>
>>> x = 1e300 * 1e300
>>> x
inf

>>> '{:g}'.format(x)
'inf'
>>> '{:g}'.format(x * 0)
'nan'

>>> '{:G}'.format(x)
'INF'
>>> '{:G}'.format(x * 0)
'NAN'

You’ll see similar behavior with the f and F presentations types as well:

>>>
>>> '{:f}'.format(x)
'inf'
>>> '{:F}'.format(x)
'INF'

>>> '{:f}'.format(x * 0)
'nan'
>>> '{:F}'.format(x * 0)
'NAN'

For more information on floating-point representation, inf, and NaN, check out the Wikipedia page on IEEE 754-1985.

The modulo operator supports r and a conversion types to force conversion by repr() and ascii(), respectively. Python’s .format() method doesn’t support r and a presentation types, but you can accomplish the same thing with the !r and !a conversion components in the replacement field.

Finally, you can use the % conversion type with the modulo operator to insert a literal '%' character into the output:

>>>
>>> '%f%%' % 65.0
'65.000000%'

You don’t need anything special to insert a literal '%' character into the Python .format() method’s output, so the % presentation type serves a different handy purpose for percent output. It multiplies the specified value by 100 and appends a percent sign:

>>>
>>> '{:%}'.format(0.65)
'65.000000%'

The remaining parts of <format_spec> indicate how the chosen presentation type is formatted, in much the same way as the string modulo operator’s width and precision specifiers and conversion flags. These are described more fully in the following sections.

The <fill> and <align> Subcomponents

<fill> and <align> control how formatted output is padded and positioned within the specified field width. These subcomponents only have meaning when the formatted field value doesn’t occupy the entire field width, which can only happen if a minimum field width is specified with <width>. If <width> isn’t specified, then <fill> and <align> are effectively ignored. You’ll cover <width> later on in this tutorial.

Here are the possible values for the <align> subcomponent:

  • <
  • >
  • ^
  • =

A value using the less than sign (<) indicates that the output is left-justified:

>>>
>>> '{0:<8s}'.format('foo')
'foo     '
>>> '{0:<8d}'.format(123)
'123     '

This behavior is the default for string values.

A value using the greater than sign (>) indicates that the output should be right-justified:

>>>
>>> '{0:>8s}'.format('foo')
'     foo'
>>> '{0:>8d}'.format(123)
'     123'

This behavior is the default for numeric values.

A value using a caret (^) indicates that the output should be centered in the output field:

>>>
>>> '{0:^8s}'.format('foo')
'  foo   '
>>> '{0:^8d}'.format(123)
'  123   '

Finally, you can also specify a value using the equals sign (=) for the <align> subcomponent. This only has meaning for numeric values, and only when a sign is included in the output.

When numeric output includes a sign, it’s normally placed directly to the left of the first digit in the number, as shown above. If <align> is set to the equals sign (=), then the sign appears at the left edge of the output field, and padding is placed in between the sign and the number:

>>>
>>> '{0:+8d}'.format(123)
'    +123'
>>> '{0:=+8d}'.format(123)
'+    123'

>>> '{0:+8d}'.format(-123)
'    -123'
>>> '{0:=+8d}'.format(-123)
'-    123'

You’ll cover the <sign> component in detail in the next section.

<fill> specifies how to fill in extra space when the formatted value doesn’t completely fill the output width. It can be any character except for curly braces ({}). (If you really feel compelled to pad a field with curly braces, then you’ll just have to find another way!)

Some examples of the use of <fill> are shown below:

>>>
>>> '{0:->8s}'.format('foo')
'-----foo'

>>> '{0:#<8d}'.format(123)
'123#####'

>>> '{0:*^8s}'.format('foo')
'**foo***'

If you specify a value for <fill>, then you should also include a value for <align> as well.

The <sign> Subcomponent

You can control whether a sign appears in numeric output with the <sign> component. For example, in the following, the plus sign (+) specified in the <format_spec> indicates that the value should always be displayed with a leading sign:

>>>
>>> '{0:+6d}'.format(123)
'  +123'
>>> '{0:+6d}'.format(-123)
'  -123'

Here, you use the plus sign (+), so a sign will always be included for both positive and negative values. If you use the minus sign (-), then only negative numeric values will include a leading sign, and positive values won’t:

>>>
>>> '{0:-6d}'.format(123)
'   123'
>>> '{0:-6d}'.format(-123)
'  -123'

When you use a single space (' '), it means a sign is included for negative values, and an ASCII space character for positive values:

>>>
>>> '{0:*> 6d}'.format(123)
'** 123'

>>> '{0:*>6d}'.format(123)
'***123'

>>> '{0:*> 6d}'.format(-123)
'**-123'

Since the space character is the default fill character, you’d only notice the effect of this if an alternate <fill> character is specified.

Lastly, recall from above that when you specify the equals sign (=) for <align> and you include a <sign> specifier, the padding goes between the sign and the value, rather than to the left of the sign.

The # Subcomponent

When you specify a hash character (#) in <format_spec>, Python will select an alternate output form for certain presentation types. This is analogous to the # conversion flag for the string modulo operator. For binary, octal, and hexadecimal presentation types, the hash character (#) causes inclusion of an explicit base indicator to the left of the value:

>>>
>>> '{0:b}, {0:#b}'.format(16)
'10000, 0b10000'
>>> '{0:o}, {0:#o}'.format(16)
'20, 0o20'
>>> '{0:x}, {0:#x}'.format(16)
'10, 0x10'

As you can see, the base indicator can be 0b, 0o, or 0x.

For floating-point or exponential presentation types, the hash character forces the output to contain a decimal point, even if the output consists of a whole number:

>>>
>>> '{0:.0f}, {0:#.0f}'.format(123)
'123, 123.'
>>> '{0:.0e}, {0:#.0e}'.format(123)
'1e+02, 1.e+02'

For any presentation type other than those shown above, the hash character (#) has no effect.

The 0 Subcomponent

If output is smaller than the indicated field width and you specify the digit zero (0) in <format_spec>, then values will be padded on the left with zeros instead of ASCII space characters:

>>>
>>> '{0:05d}'.format(123)
'00123'
>>> '{0:08.1f}'.format(12.3)
'000012.3'

You’ll typically use this for numeric values, as shown above. However, it works for string values as well:

>>>
>>> '{0:>06s}'.format('foo')
'000foo'

If you specify both <fill> and <align>, then <fill> overrides 0:

>>>
>>> '{0:*>05d}'.format(123)
'**123'

<fill> and 0 essentially control the same thing, so there really isn’t any need to specify both. In fact, 0 is really superfluous, and was probably included as a convenience for developers who are familiar with the string modulo operator’s similar 0 conversion flag.

The <width> Subcomponent

<width> specifies the minimum width of the output field:

>>>
>>> '{0:8s}'.format('foo')
'foo     '
>>> '{0:8d}'.format(123)
'     123'

Note that this is a minimum field width. Suppose you specify a value that’s longer than the minimum:

>>>
>>> '{0:2s}'.format('foobar')
'foobar'

In this case, <width> is effectively ignored.

The <group> Subcomponent

<group> allows you to include a grouping separator character in numeric output. For decimal and floating-point presentation types, <group> may be specified as either a comma character (,) or an underscore character (_). That character then separates each group of three digits in the output:

>>>
>>> '{0:,d}'.format(1234567)
'1,234,567'
>>> '{0:_d}'.format(1234567)
'1_234_567'

>>> '{0:,.2f}'.format(1234567.89)
'1,234,567.89'
>>> '{0:_.2f}'.format(1234567.89)
'1_234_567.89'

A <group> value using an underscore (_) may also be specified with the binary, octal, and hexadecimal presentation types. In that case, each group of four digits is separated by an underscore character in the output:

>>>
>>> '{0:_b}'.format(0b111010100001)
'1110_1010_0001'
>>> '{0:#_b}'.format(0b111010100001)
'0b1110_1010_0001'

>>> '{0:_x}'.format(0xae123fcc8ab2)
'ae12_3fcc_8ab2'
>>> '{0:#_x}'.format(0xae123fcc8ab2)
'0xae12_3fcc_8ab2'

If you try to specify <group> with any presentation type other than those listed above, then your code will raise an exception.

The .<prec> Subcomponent

.<prec> specifies the number of digits after the decimal point for floating point presentation types:

>>>
>>> '{0:8.2f}'.format(1234.5678)
' 1234.57'
>>> '{0:8.4f}'.format(1.23)
'  1.2300'

>>> '{0:8.2e}'.format(1234.5678)
'1.23e+03'
>>> '{0:8.4e}'.format(1.23)
'1.2300e+00'

For string types, .<prec> specifies the maximum width of the converted output:

>>>
>>> '{:.4s}'.format('foobar')
'foob'

If the output would be longer than the value specified, then it will be truncated.

The String .format() Method: Nested Replacement Fields

Recall that you can specify either <width> or <prec> by an asterisk with the string modulo operator:

>>>
>>> w = 10
>>> p = 2
>>> '%*.*f' % (w, p, 123.456)  # Width is 10, precision is 2
'    123.46'

The associated values are then taken from the argument list. This allows <width> and <prec> to be evaluated dynamically at run-time, as shown in the example above. Python’s .format() method provides similar capability using nested replacement fields.

Inside a replacement field, you can specify a nested set of curly braces ({}) that contains a name or number referring to one of the method’s positional or keyword arguments. That portion of the replacement field will then be evaluated at run-time and replaced using the corresponding argument. You can accomplish the same effect as the above string modulo operator example with nested replacement fields:

>>>
>>> w = 10
>>> p = 2
>>> '{2:{0}.{1}f}'.format(w, p, 123.456)
'    123.46'

Here, the <name> component of the replacement field is 2, which indicates the third positional parameter whose value is 123.456. This is the value to be formatted. The nested replacement fields {0} and {1} correspond to the first and second positional parameters, w and p. These occupy the <width> and <prec> locations in <format_spec> and allow field width and precision to be evaluated at run-time.

You can use keyword arguments with nested replacement fields as well. This example is functionally equivalent to the previous one:

>>>
>>> w = 10
>>> p = 2
>>> '{val:{wid}.{pr}f}'.format(wid=w, pr=p, val=123.456)
'    123.46'

In either case, the values of w and p are evaluated at run-time and used to modify the <format_spec>. The result is effectively the same as this:

>>>
>>> '{0:10.2f}'.format(123.456)
'    123.46'

The string modulo operator only allows <width> and <prec> to be evaluated at run-time in this way. By contrast, with Python’s .format() method you can specify any portion of <format_spec> using nested replacement fields.

In the following example, the presentation type <type> is specified by a nested replacement field and determined dynamically:

>>>
>>> bin(10), oct(10), hex(10)
('0b1010', '0o12', '0xa')
>>> for t in ('b', 'o', 'x'):
...     print('{0:#{type}}'.format(10, type=t))
...
0b1010
0o12
0xa

Here, the grouping character <group> is nested:

>>>
>>> '{0:{grp}d}'.format(123456789, grp='_')
'123_456_789'
>>> '{0:{grp}d}'.format(123456789, grp=',')
'123,456,789'

Whew! That was an adventure. The specification of the template string is virtually a language unto itself!

As you can see, string formatting can be very finely tuned when you use Python’s .format() method. Next, you’ll see one more technique for string and output formatting that affords all the advantages of .format(), but with more direct syntax.

The Python Formatted String Literal (f-String)

In version 3.6, a new Python string formatting syntax was introduced, called the formatted string literal. These are also informally called f-strings, a term that was initially coined in PEP 498, where they were first proposed.

f-String Syntax

An f-string looks very much like a typical Python string except that it’s prepended by the character f:

>>>
>>> f'foo bar baz'
'foo bar baz'

You can also use an uppercase F:

>>>
>>> s = F'qux quux'
>>> s
'qux quux'

The effect is exactly the same. Just like with any other type of string, you can use single, double, or triple quotes to define an f-string:

>>>
>>> f'foo'
'foo'
>>> f"bar"
'bar'
>>> f'''baz'''
'baz'

The magic of f-strings is that you can embed Python expressions directly inside them. Any portion of an f-string that’s enclosed in curly braces ({}) is treated as an expression. The expression is evaluated and converted to string representation, and the result is interpolated into the original string in that location:

>>>
>>> s = 'bar'
>>> print(f'foo.{s}.baz')
foo.bar.baz

The interpreter treats the remainder of the f-string—anything not inside curly braces—just as it would an ordinary string. For example, escape sequences are processed as expected:

>>>
>>> s = 'bar'
>>> print(f'foo\n{s}\nbaz')
foo
bar
baz

Here’s the example from earlier using an f-string:

>>>
>>> quantity = 6
>>> item = 'bananas'
>>> price = 1.74
>>> print(f'{quantity} {item} cost ${price}')
6 bananas cost $1.74

This is equivalent to the following:

>>>
>>> quantity = 6
>>> item = 'bananas'
>>> price = 1.74
>>> print('{0} {1} cost ${2}'.format(quantity, item, price))
6 bananas cost $1.74

Expressions embedded in f-strings can be almost arbitrarily complex. The examples below show some of the possibilities:

  • Variables:

    >>>
    >>> quantity, item, price = 6, 'bananas', 1.74
    >>> f'{quantity} {item} cost ${price}'
    '6 bananas cost $1.74'
    
  • Arithmetic expressions:

    >>>
    >>> quantity, item, price = 6, 'bananas', 1.74
    >>> print(f'Price per item is ${price/quantity}')
    Price per item is $0.29
    
    >>> x = 6
    >>> print(f'{x} cubed is {x**3}')
    6 cubed is 216
    
  • Objects of composite types:

    >>>
    >>> a = ['foo', 'bar', 'baz']
    >>> d = {'foo': 1, 'bar': 2}
    
    >>> print(f'a = {a} | d = {d}')
    a = ['foo', 'bar', 'baz'] | d = {'foo': 1, 'bar': 2}
    
  • Indexing, slicing, and key references:

    >>>
    >>> a = ['foo', 'bar', 'baz']
    >>> d = {'foo': 1, 'bar': 2}
    
    >>> print(f'First item in list a = {a[0]}')
    First item in list a = foo
    
    >>> print(f'Last two items in list a = {a[-2:]}')
    Last two items in list a = ['bar', 'baz']
    
    >>> print(f'List a reversed = {a[::-1]}')
    List a reversed = ['baz', 'bar', 'foo']
    
    >>> print(f"Dict value for key 'bar' is {d['bar']}")
    Dict value for key 'bar' is 2
    
  • Function and method calls:

    >>>
    >>> a = ['foo', 'bar', 'baz', 'qux', 'quux']
    >>> print(f'List a has {len(a)} items')
    List a has 5 items
    
    >>> s = 'foobar'
    >>> f'--- {s.upper()} ---'
    '--- FOOBAR ---'
    
    >>> d = {'foo': 1, 'bar': 2}
    >>> print(f"Dict value for key 'bar' is {d.get('bar')}")
    Dict value for key 'bar' is 2
    
  • Conditional expressions:

    >>>
    >>> x = 3
    >>> y = 7
    >>> print(f'The larger of {x} and {y} is {x if x > y else y}')
    The larger of 3 and 7 is 7
    
    >>> age = 13
    >>> f'I am {"a minor" if age < 18 else "an adult"}.'
    'I am a minor.'
    
  • Object attributes:

    >>>
    >>> z = 3+5j
    >>> z
    (3+5j)
    
    >>> print(f'real = {z.real}, imag = {z.imag}')
    real = 3.0, imag = 5.0
    

To include a literal curly brace in an f-string, escape it by doubling it, the same as you would in a template string for Python’s .format() method:

>>>
>>> z = 'foobar'
>>> f'{{ {z[::-1]} }}'
'{ raboof }'

You may prefix an f-string with 'r' or 'R' to indicate that it is a raw f-string. In that case, backslash sequences are left intact, just like with an ordinary string:

>>>
>>> z = 'bar'
>>> print(f'foo\n{z}\nbaz')
foo
bar
baz
>>> print(rf'foo\n{z}\nbaz')
foo\nbar\nbaz
>>> print(fr'foo\n{z}\nbaz')
foo\nbar\nbaz

Note that you can specify the 'r' first and then the 'f', or vice-versa.

f-String Expression Limitations

There are a few minor restrictions on f-string expression. These aren’t too limiting, but you should know what they are. First, an f-string expression can’t be empty:

>>>
>>> f'foo{}bar'
  File "<stdin>", line 1
SyntaxError: f-string: empty expression not allowed

It isn’t obvious why you’d want to do this. But if you feel compelled to try, then just know that it won’t work.

Additionally, an f-string expression can’t contain a backslash (\) character. Among other things, that means you can’t use a backslash escape sequence in an f-string expression:

>>>
>>> print(f'foo{\n}bar')
  File "<stdin>", line 1
SyntaxError: f-string expression part cannot include a backslash
>>> print(f'foo{\'}bar')
  File "<stdin>", line 1
SyntaxError: f-string expression part cannot include a backslash

You can get around this limitation by creating a temporary variable that contains the escape sequence you want to insert:

>>>
>>> nl = '\n'
>>> print(f'foo{nl}bar')
foo
bar
>>> quote = '\''
>>> print(f'foo{quote}bar')
foo'bar

Lastly, an expression in an f-string that is triple-quoted can’t contain comments:

>>>
>>> z = 'bar'

>>> print(f'''foo{
... z
... }baz''')
foobarbaz

>>> print(f'''foo{
... z    # Comment
... }''')
  File "<stdin>", line 3
SyntaxError: f-string expression part cannot include '#'

Note, however, that the multiline f-string may contain embedded newlines.

f-String Formatting

Like Python’s .format() method, f-strings support extensive modifiers that control the final appearance of the output string. There’s more good news, too. If you’ve mastered the Python .format() method, then you already know how to use Python to format f-strings!

Expressions in f-strings can be modified by a <conversion> or <format_spec>, just like replacement fields used in the .format() template. The syntax is identical. In fact, in both cases Python will format the replacement field using the same internal function. In the following example, !r is specified as a <conversion> component in the .format() template string:

>>>
>>> s = 'foo'
>>> '{0!r}'.format(s)
"'foo'"

This forces conversion to be performed by repr(). You can get essentially the same code using an f-string instead:

>>>
>>> s = 'foo'
>>> f'{s!r}'
"'foo'"

All the <format_spec> components that work with .format() also work with f-strings:

>>>
>>> n = 123
>>> '{:=+8}'.format(n)
'+    123'
>>> f'{n:=+8}'
'+    123'

>>> s = 'foo'
>>> '{0:*^8}'.format(s)
'**foo***'
>>> f'{s:*^8}'
'**foo***'

>>> n = 0b111010100001
>>> '{0:#_b}'.format(n)
'0b1110_1010_0001'
>>> f'{n:#_b}'
'0b1110_1010_0001'

Nesting works as well, like nested replacement fields with Python’s .format() method:

>>>
>>> a = ['foo', 'bar', 'baz', 'qux', 'quux']
>>> w = 4
>>> f'{len(a):0{w}d}'
'0005'

>>> n = 123456789
>>> sep = '_'
>>> f'{(n * n):{sep}d}'
'15_241_578_750_190_521'

This means formatting items can be evaluated at run-time.

f-strings and Python’s .format() method are, more or less, two different ways of doing the same thing, with f-strings being a more concise shorthand. The following expressions are essentially the same:

f'{<expr>!<conversion>:<format_spec>}'

'{0!<conversion>:<format_spec>}'.format(<expr>)

If you want to explore f-strings further, then check out Python 3’s f-Strings: An Improved String Formatting Syntax (Course).

Conclusion

In this tutorial, you mastered two additional techniques that you can use in Python to format string data. You should now have all the tools you need to prepare string data for output or display!

You might be wondering which Python formatting technique you should use. Under what circumstances would you choose .format() over the f-string? See Python String Formatting Best Practices for some considerations to take into account.

In the next tutorial, you’re going to learn more about functions in Python. Throughout this tutorial series, you’ve seen many examples of Python’s built-in functions. In Python, as in most programming languages, you can define your own custom user-defined functions as well. If you can’t wait to learn how then continue on to the next tutorial!


[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Stories in My Pocket: Refactoring and asking for forgiveness(14 hours, 41 minutes ago)

Recently, I had a great interaction with one of my coworkers that I think is worth sharing, with the hope you may learn a bit about refactoring and python.

My colleague came to me to help him think through a problem that surfaced with a change to a project. The code in question sends a file to a remote storage service. It looked like this:


Read more...

Mike Driscoll: PyDev of the Week: Martin Fitzpatrick(22 hours, 23 minutes ago)

This week we welcome Martin Fitzpatrick (@mfitzp) as our PyDev of the Week! Martin is the author of “Create Simple GUI Applications with Python and Qt 5” and the creator of the LearnPyQt website. You can also check out his personal site or see what he’s up to by visiting his Github profile. Let’s spend some time getting to know Martin better!

Martin Fitzpatrick

Can you tell us a little about yourself (hobbies, education, etc):

I’m a developer from the United Kingdom, who’s been working with Python for the past 12 years, and living in the Netherlands (Amersfoort) for the past 5.

I started coding on 8 bit machines back in the early 90s, creating platform games of dubious quality — in my defence we didn’t have StackOverflow back then. Later I moved onto the PC, first writing DOS games and then, after someone invented the internet, doing a stint of web dev. I’ve been programming on and off ever since.

Rather than pursue software development as a career, I instead took a long detour into healthcare/biology. I worked first in the ambulance service, then as a physiotherapy assistant and finally completed a degree and PhD in Bioinformatics & Immunology. This last step was where I discovered Python, ultimately leading me to where I am now.

In my spare time I tinker in my workshop, creating daft electronic games and robots.

I like robots.

Why did you start using Python?

I first used Python back in 2007 when I was looking for at alternatives to building websites with Drupal/PHP. That led me to Django and Python. It felt so much simpler and more logical than what I’d used before, after knocking something together in an afternoon I was basically hooked.

For the next few years I was using Python almost exclusively for web development, and it probably would have stayed that way was it not for my PhD.

My thesis project was looking at the effects of metabolism on rheumatoid arthritis, and required me to analyse some big chunks of data. Having worked with Python for the previous 4 years it only seemed natural to try and use it here, rather than stop and learn R or MATLAB. The Python data analysis landscape was still a bit rough back then, but improving quickly — pandas and Jupyter notebooks first appeared during this time. The final couple of years of my PhD I was looking to make the tools I’d written more accessible to non-technical users and started building GUI applications with PyQt5.

In the past couple of years I discovered microcontrollers (ESP8266 and Raspberry Pi) and have built some silly things with MicroPython.

What other programming languages do you know and which is your favorite?

Python is my favourite, hands down. There is something about the language that lines up very well with my brain, might be all the empty space.

I have learnt and forgotten quite a few languages including PHP, Pascal, Perl, Prolog and Z80 assembler. I can still bash something together in C and does MicroPython count as another language?

What projects are you working on now?

For the past month I’ve been working on updates to my PyQt5 book, Create Simple GUI Applications with Python & Qt and the accompanying PyQt5/PySide2 tutorials. There are plenty of good beginner resources available but once you get into developing real applications you’re a bit out of luck. I’ve now got a small team of writers together now which is really kicking things up a notch, it’s a lot of fun.

The site itself is custom built on top of Wagtail/Django, and I would like to make that available to other Python developers who want to host their own courses and tutorials.

Over on my workbench I have a few electronic game projects in progress using Python but can’t share anything yet. My biggest success to date was Etch-A-Snap a Raspberry Pi powered Etch-A-Sketch camera. Basically, you take a photo and it draws it automatically on a mini Etch-A-Sketch. That was bought by a Japanese collector which was kind of mind-blowing.

If anyone is interested in working with me I’m always open to collaborate!

Which Python libraries are your favorite (core or 3rd party)?

Python numeric libraries (numpy, scipy, pandas) and Jupyter notebooks are the big standouts for me. Collectively they have saved me hours upon hours of work, and having these kinds of tools available has been a major boost to Python’s popularity.

I think I have to say PyQt5/PySide2 here as well.

How did you become a self-published author?

When I started writing GUI applications in PyQt5 it was a bit of an uphill struggle. There were very few examples or tutorials available at the time, so I was stuck converting examples from C++. Once I had it figured out I just sat and wrote down everything I’d needed to learn on the way — hopefully to save other people from the same fate. That was the first edition of the book, and I’ve been updating it regularly ever since.

Self publishing was just the simplest option to begin with, allowing me to concentrate on writing the book I wanted to read. Offers I have had from publishers all come with changes or restrictions on distribution I don’t agree with (the book is currently Creative Commons licensed).

Why did you choose PyQt over the other Python GUI toolkits?

During my PhD I was developing a tool for visualizing chemical pathways in cells and needed a good way to visualize overlaid pathways. The only solution I could find for this involved rendering using HTML/Javascript in an embedded window.

I built prototypes of the app with Tkinter, wxWidgets and Qt, and actually quite liked wxWidgets for its native widgets. But Qt was the only toolkit which shipped with its own built-in browser and so worked reliably across different platforms (I was on MacOS, my supervisor on Windows).

The rest, as they say, is history. The big benefit of Qt for me now is the QGraphicsScene vector graphics canvas which is great for interactive visualizations.

Do you have any advice for other potential authors?

Firstly, everybody knows something, or some combination of things or some things from a certain angle that is unique enough to form the basis of a book. Even if a book already exists in your niche, your own voice or experience is valuable.

Secondly, pick a topic that you’re interested in but that you also enjoy doing. Writing a programming book means writing a lot of example code, and if you don’t enjoy creating these things it’s going to be a unpleasant experience.

One of my first steps writing my book was to start a collection of example PyQt5 applications including clones of classics like Paint and Solitaire. This helped me figure out what was *required* knowledge, but — more importantly — it was a lot of fun.

Is there anything else you’d like to say?

I often get questions from Python developers who’ve come to programming a little later, asking “am I too old to start doing this professionally?”

The answer is no! I’ve had a career spanning admin (finance, human resources), healthcare (ambulance service, physiotherapy assistant, disability support worker), research scientist (PhD and post-doc) and landed my first actual “programming job” at 37.

One great advantage of learning to code in Python is that it can be applied to so many different problem domains. Whether you build your experience automating some office reports, doing data analysis or building websites, those skills will transfer and and your Python will improve.

Thanks for doing the interview, Martin!

The post PyDev of the Week: Martin Fitzpatrick appeared first on The Mouse Vs. The Python.

Anwesha Das: February PyLadies Pune workshop(22 hours, 44 minutes ago)

It was the time for “learning Python with harware” in February, 2020 with PyLadies in Pune. Coding in Python becomes fun when one can see the changes it makes in the hardware.

Selecting a place for work is always a difficult task as any organizer. College Of Engineering Pune (COEP) has always been supportive of PyLadies Pune. When I approached Abhijit for the venue he readily agreed. My sincere gratitude to him, Women Engineers Group and the FOSSMeet Pune team enough for that.

Once I reached the venue it was already a full house and still people were coming in. We had more than 55 students of 1st to 3rd year, attending the workshop. The first year students already knew Python. Around 12-14 people were writing Python for the first time.

The workshop started with the very basics of the language on the terminal.

feb1pyladies

Then came the exciting part, trying Python on hardware. We were using Circuit Playground Express boards by Adafruit. Nina and Kattni provided these boards to us.. We, people on the other side of the world, do not have easy access to Adafruit hardwares. It takes a lot of time and money to get them. The students were holding the any such board for the first time. I can not thank Nina and Kattni & Adafruit enough for that.

We started with blinking the first LED of the board. When the students lit their first LED the smile and light in their eyes were precious :). Following that we spend some time with the simple codes. We tried our hands on different modules of Circuit Python. We took the help from the tutorial provided in Adafruit website. The students were enjoying and indulged into creativity. So I decided to give them problem statements instead of showing them code. I was happy to see how fast they were solving it and experimenting with different patterns, colours.
The workshop finished too soon than I expected. We bid a good bye to them with a promise to return with such a workshop like this.

feb2pyladies

This was the first time for me to take hardware workshop for a larger group. I was scared at the beginning. Therefore I spent many days preparing for it. But I learned a lot new things in there :

A workshop can not be planned. One might have structure but not a plan. All of it depends on the participants and how they respond. And of course on the mercy of the hardware. I was amazed to see the problems that came up. Be it the :

  • code.py becoming read only;
  • how different approaches could be there to solve one problem;
  • People writing Python like C for college :)

The post will be incomplete if I do not thank [Kushal]9https://twitter.com/kushaldas) for being a patient Teaching Assistant for my workshop and helping the students throughout.

It was a great experience for me. The feedback of a student that “I was sure that I am going to be bored. But you taught so well that 3 and half hours just flew”, will be on my inspirational board from now on :).

Robin Parmar: A review of Processing books(1 day, 3 hours ago)

Processing is the free and open Java development environment that targets artists who are intrigued by generative code. In essence it is the Java language with a friendly development interface and built-in libraries to get you started.

There are plenty of ways to learn Processing, including the tutorials on the organisation's website, and the built-in examples that come with the distribution. But if you prefer a printed book, keep reading. This article will review nine available publications, so you can make an informed purchase decision.

For the sake of completeness I will also append information on two books I haven't had a chance to read.

Recommended books

Reas, Casey and Ben Fry. 2014. Processing: A Programming Handbook for Visual Designers and Artists. Second edition. London: The MIT Press. website

This book is straight from the horse's mouth; Reas and Fry created Processing. The 38 chapters cover an extensive array of topics: drawing and shapes, interactivity, variables and program flow, typography, images and transformations, animation, data manipulation, 3D graphics and rendering. The four main sections are bookended by informative practitioner interviews. There's a glossary, reading list, and index. If 640 pages isn't enough, supplementary chapters are available on the website. If you had to buy just one Processing book, this would be the one, since it covers both the basics and more advanced applications.


Shiffman, Daniel. 2015. Learning Processing: A Beginner’s Guide to Programming Images, Animation, and Interaction. Second edition. Burlington, MA: Morgan Kaufmann. website

If you have done little previous coding, this is a good guide, since it approaches Processing from scratch. The thorough topic list includes language constructs (loops, functions, objects), maths and transformations, working with images and video, data manipulation and networking, sound, and exporting. 525 pages.

Many of the examples will be familiar from the tutorials on the Processing website. Shiffman is also known for his cheerleader-style videos, which cover just about every topic under the sun with a ridiculous amount of enthusiasm. If you avail of the resources Shiffman's generosity makes available for free, you might find little need for this book. But for those who prefer the printed page, it's a good alternative to Reas and Fry.


Shiffman, Daniel. 2012. The Nature of Code. New York: self-published. website

This book is a more advanced consideration of techniques for generating algorithmic visuals. It's the perfect follow-up to Shiffman's Learning Processing. Indeed, those who already have some coding background should skip directly to this volume, after looking at the online tutorials.

The first sections cover physics (vectors, forces, oscillation) and particle systems. With this groundwork in hand, topics in algorithmic visuals are tackled, including autonomous agents, cellular automata, fractals, genetic algorithms, and neural networks. All of this in Shiffman's usual style: pleasant, accurate, thorough. I would buy this alongside Reas and Fry, since there's not much overlap in the topics. 520 pages.


Fry, Ben. 2008. Visualizing Data. Sebastopol, California: O'Reilly. publisher and author websites

This book provides a rigorous methodological framework for approaching the complex topic of data visualisation, alongside useful code examples. Two introductory chapters cover the visualisation workflow and the basics of Processing. Then we launch into the nitty-gritty of acquiring and parsing data, mapping, providing interaction, time series, scatterplots, tree maps, recursion, networks and graphs, etc. You will notice immediately that this book is more focused and technical than others. An updated edition would ice the cake, since this volume is getting rather old. 365 pages including index.


Greenberg, Ira; Dianna Xu; Deepak Kumar. 2013. Processing: Creative Coding and Generative Art in Processing 2. New York: Apress. website

This book provides an introduction to Processing for those new to the environment, teaching programming fundamentals and the elements of the language alongside sections tackling specific problems. There is nothing wrong with this approach, but if you already own one of the other introductions, the first half of this book is redundant. Thankfully, the topics do get more involved: advanced OOP (encapsulation, inheritance), data visualisation (parsing, graphs, heat map, word clouds, interaction, tree map), motion (vectors, boundary conditions, verlet integration), recursion and L-systems, image manipulation (bitwise manipulation, masking, filtering, convolution), 3D graphics (perspective, projection). 445 pages including index. While the contents overlap others here, these authors offer a deep and broad approach of significant value.


Other titles

Richardson, Andrew. 2016. Data-driven Graphic Design: Creative Coding For Visual Communications. London: Bloomsbury. website

This is an odd book that doesn't seem to know its audience. In part, it's a coffee-table volume highlighting creative use of code with full-page illustrations. Such a book should present interviews with practitioners, detailed background on artworks, and analysis of the intriguing interactions between technology, aesthetics, and the marketplace. But very little of this is actually found between these covers.

Instead, each chapter concludes with code examples, as though this is a book for programmers. But coders are hardly edified by statements like "The ability of a computer program to infinitely repeat calculations and processes gives it an enormous potential for creating complex drawings and graphics." This obvious statement is found, not in the preface, but well into chapter two! Topics include generative drawing, growth and form, dynamic typography, interactive projections, and data visualisation.


Pearson, Matt. 2011. Generative Art. Shelter Island, NY: Manning. website

Similar to the Richardson volume, this book attempts an overview of generative art while at the same time introducing the coding skills that might produce such wonders. With its greater emphasis on programming, it does a better job. But at 200 pages, the volume is slim. The first chapter provides a sketchy (pun intended) introduction to the topic, with insufficient historical examples. The second chapter contains a very basic introduction to Processing. Further topics include: randomness and Perlin noise, simple shapes, OOP, cellular automata, and fractals. The code examples reward study, and are the strength of this volume. But this material is covered with greater focus elsewhere.


Lees, Antony, ed. 2019. Beginning Graphics Programming with Processing 3. self-published. website

This book was assembled as an independent publishing project by a group of students. Section 1 covers programming principles (algorithms and operators through to OOP); section 2 examines shapes and interaction; section 3 covers images, rendering, curves, and 3D. These contents are most suitable for a beginner; hence the book competes head-on with Shiffman. 650 pages.


Gradwohl, Nikolaus. 2013. Processing 2: Creative Coding Hotshot. Birmingham, UK: Packt Publishing.

This book consists of nine projects: cardboard robots re-enacting Romeo and Juliet, a sound-reactive dance floor, a moon lander game for Android, etc. These emphasise interactivity and physical computing, integrating Processing with environments such as Arduino. You'll want to get this if you are tasked with running a workshop for tweens, but otherwise it's rather difficult to know the audience.


And more

Two further books come highly recommended, but I haven't yet had the opportunity to check them out. Send me a copy if you'd like a review!

Bohnacker, Hartmut; Benedikt Gross; Julia Laub; Claudius Lazzeroni. 2012. Generative Design: Visualize, Program, and Create with Processing. Hudson, New York: Princeton Architectural Press. website

Don't confuse this with the newer version that covers similar topics, but implements them in Javascript rather than Processing. 472 pages.


Glassner, Andrew. 2011. Processing for Visual Artists: How to Create Expressive Images and Interactive Art. Natick, MA: A. K. Peters. website

Designed for code newcomers, this includes advice on workflow and standards not commonly found in other books. An enormous 937 pages including index. Freakishly expensive.

Glyph Lefkowitz: Modularity for Maintenance(1 day, 4 hours ago)

Never send a human to do a machine’s job.

One of the best things about maintaining open source in the modern era is that there are so many wonderful, free tools to let machines take care of the busy-work associated with collaboration, code-hosting, continuous integration, code quality maintenance, and so on.

There are lots of great resources that explain how to automate various things that make maintenance easier.

Here are some things you can configure your Python project to do:

  1. Continuous integration, using any one of a number of providers:
    1. GitHub Actions
    2. CircleCI
    3. Azure Pipelines
    4. Appveyor
    5. GitLab CI&CD
    6. Travis CI
  2. Separate multiple test jobs with tox
  3. Lint your code with flake8
  4. Type-Check your code with MyPy
  5. Auto-update your dependencies, with one of:
    1. pyup.io
    2. requires.io, or
    3. Dependabot
  6. automatically find common security issues with Bandit
  7. check the status of your code coverage, with:
    1. Coveralls, or
    2. Codecov
  8. Auto-format your code with:
    1. Black for style
    2. autopep8 to fix common errors
    3. isort to keep your imports tidy
  9. Automatically update your dependencies
  10. Help your developers remember to do all of those steps with pre-commit
  11. Automatically release your code to PyPI via your CI provider
    1. including automatically building any C code for multiple platforms as a wheel so your users won’t have to
    2. and checking those build artifacts:
    3. to make sure they include all the files they should, with check-manifest
    4. and also that the binary artifacts have the correct dependencies for Linux
    5. and also for macOS
  12. Organize your release notes and versioning with towncrier

All of these tools are wonderful.

But... let’s say you1 maintain a few dozen Python projects. Being a good maintainer, you’ve started splitting up your big monolithic packages into smaller ones, so your utility modules can be commonly shared as widely as possible rather than re-implemented once for each big frameworks. This is great!

However, every one of those numbered list items above is now a task per project that you have to repeat from scratch. So imagine a matrix with all of those down one side and dozens of projects across the top - the full Cartesian product of these little administrative tasks is a tedious and exhausting pile of work.

If you’re lucky enough to start every project close to perfect already, you can skip some of this work, but that partially just front-loads the tedium; plus, projects tend to start quite simple, then gradually escalate in complexity, so it’s helpful to be able to apply these incremental improvements one at a time, as your project gets bigger.

I really wish there were a tool that could take each of these steps and turn them into a quick command-line operation; like, I type pyautomate pypi-upload and the tool notices which CI provider I use, whether I use tox or not, and adds the appropriate configuration entries to both my CI and tox configuration to allow me to do that, possibly prompting me for a secret. Same for pyautomate code-coverage or what have you. All of these automations are fairly straightforward; almost all of the files you need to edit are easily parse-able either as yaml, toml, or ConfigParser2 files.

A few years ago, I asked for this to be added to CookieCutter, but I think the task is just too big and complicated to reasonably expect the existing maintainers to ever get around to it.

If you have a bunch of spare time, and really wanted to turbo-charge the Python open source community, eliminating tons of drag on already-over-committed maintainers, such a tool would be amazing.


  1. and by you, obviously, I mean “I” 

  2. “INI-like files”, I guess? what is this format even called? 

Roberto Alsina: Airflow By Example(1 day, 8 hours ago)

Apache Airflow is a very interesting, popular and free tool to create, manage and monitor workflows, for example if you want to do ETL (Extract / Transform / Load) on data.

This sort of enterprise software often may seem complicated or overly unrelated to our everyday experience as developers but ... is it, really? How about if I just want to watch some TV shows? And experiment with some enterprise-level software at the same time?

Let's do that by learning how to use Airflow to watch TV.


Caveat: This post was originally a twitter thread, that's why all the examples are images and you can't copy/paste them. But hey, at least they are short. Also, typos, because I really just did this while tweeting it, no preparation beforehand.

Just in case: I did not download any "Star Trek: Picard" espisodes, and I have a Prime video subscription, so I don't need to download them via torrent. OTOH, if Sir Patrick ever reads this (which he won't): good job, sir!


A thread by Roberto Alsina

This is a script that gives you the information about the latest already aired episode of a TV series.

And this is a script that gives you the link to a torrent to download that episode.

This, on the other hand, is a script to download that torrent.

Of course, one thing about this script is not like the other scripts.

While the others take a couple of seconds to run, this one may take hours or days. But let's ignore that for now. This is a script that moves your downloaded file into a nice directory hierarchy.

However, I think this is nicer because it does the same thing but with names "guaranteed" to be right, more uniform, and transcodes it to, say, something a chromecast would like.

I could add extra tiny scripts that get subtitles for the languages you like and put them in the right location, and so on, but you get the idea.

Basically: it's easy to automate "I want to watch the latest episode of Picard"

However, it's totally impractical, because:

1) You have to go and tell it to get it
2) It takes hours
3) It may fail, and then you have to do it again
4) It will take hours again

But what if there was a way to define this set of tasks so that they run automatically, you know when they are working, when they start and when they finish, and you have to do nothing except waiting for a message in telegram that tells you "go watch Picard"? And that's where Apache Airflow enters the picture.

If you have a script that is a number of steps which depend on one another and are executed in order then it's possible to convert them into very simple airflow DAGs. Now I will take a little while to learn exactly HOW to do that and will continue this thread in a bit. Because, really, ETL (Extract / Transform / Load) is not as complicated as it may appear to be in most cases. BTW, if you want to have airflow with Python 3.8 (because it's nicer)

Now, this may look complicated, but really, I am defining a DAG (Directed Acyclic Graph) or "thingies connected with arrows that has no loops in it"

What are the "thingies" inside an airflow DAG? Operators. They come in many flavors, but I am using this one now, which sets up a venv, runs a function, then removes everything. Nice and clean. (airflow.apache.org/docs/stable/_a…) So, let's convert this into an Airflow operator.

It's not hard! operators are (in the case of python operators) simply functions.

Details: do all the imports inside the function.

Have a list of requirements ready if you require things.

Now I need to put this operator inside the DAG I created earlier. Again, it's a matter of declaring things.

So, now that we have a (very stupid, one node, no arrows) DAG ... what can we do with it?

Well, we can make sure airflow sees it (you need to tell airflow where your dags live)

We can check that our DAG has some task in it!

A task is just an instance of an Operator, we added one, so there should be one.

And of course, we can TEST the thing, and have our task do its trick. Note that I have to pass a date. We could even use that in a 0.0.2 version to check episodes we missed!

Hey, that worked! (BTW, of course it did not work THE FIRST TIME, come on).

Backfill means "start at this date and run whatever would have run if we had actually started at that date"

Now, if you run "airflow scheduler" and "airflow webserver" you can see things like this

And yes, that means this task will run daily and report everything in the nice web UI and all that.

But of course a single task is a lame DAG, so let's make it a bit more interesting. Now, let's create a second operator, which will run AFTER the one we had done is finished, and use its output.

It's based on this:

Following the same mechanical changes as before (imports in the function, etc) it will look like this:

This uses two pieces of data from the previous task.
So, we need to do 2 things:

1) Connect the two operators
2) Pass data from one to the other

Connecting two tasks in a DAG is simple. Declare them both and tell airflow they are connected and in what direction.

And of course that's now reflected in the airflow UI. Here you can see that Check_Picard has been successful (dark green border) and Search_Torrent has no status because it never ran (white border)

It's probably worth mentioning that patience is important at this point in the project, since "it runs quickly with immediate feedback" is not one of the benefits we are getting here.

This will be slower than just running the scripts by hand. And now we have the search_torrent task failing.

Why?

Well, luckily we are using airflow, so we have logs!

The problem is, Search_Torrent is not getting the right arguments. It "wants" a dict with at least series_name, season and episode in it.

And ... that's now how these things work in airflow :-)

Slight detour, I just ran into this: (issues.apache.org/jira/browse/AI…)

So, I need to rewrite my nice virtualenved operators into uglier not-virtualenved ones.

Shame, airflow, shame. BTW, this is a minor but important part of developing software. Sometimes, you are doing things right and it will not work because there is a bug somewhere else.

Suck it up! So, back to the code, let's recap. I now have two operators. One operator looks for the latest episode of a TV series (example: Picard), and returns all the relevant data in a serializable thing, like a dict of dicts of strings and ints.

The second one will search for a torrent of a given episode of a series. It uses the series name, the season number, and the episode number.

How does it know the data it needs to search? Since it was returned by the previous task in the DAG, it gets it from "context". Specifically it's there as a XCOM with key "return value".

And once it has all the interesting torrent information, then it just adds it to the data it got and returns THAT

How do we use these operators in our DAG?

To search for Picard, I use my tvdb operator with an argument "Picard"

To get the picard torrent, I use my rarbg operator, and use provide_context-True, so it can access the output of the other operator.

And then I hook them up

Does it work? Yes it does!

So, let's make it more intersting. A DAG of 2 nodes with one arrow is not the least interesting DAG ever but it's close!

So, what happens if I also want to watch ... "The Rookie"? I can just setup a new tvdb operator with a different argument, and connect them both to the operator that searches for torrents.

In the airflow graph it looks like this:

And in another view, the tree view, it looks like this:

So, this DAG will look every day for new episodes of Picard or The Rookie, then, if there is one, will trigger a torrent search for it. Adding further operators and tasks to actually download the torrent, transcode it, putting it in the right place, getting subtitles, and so ... is left as an exercise for the reader (I gave some hints at the beginning of the thread)

If you enjoyed this thread, consider hiring me ;-)

Senior Python Dev, eng mgmt experience, remote preferred, based near Buenos Aires, Argentina. (ralsina.me/weblog/posts/l…)

Kushal Das: No summer training 2020(1 day, 11 hours ago)

No summer training 2020 for me. Last year’s batch was beyond my capability to handle. Most of the participants did not follow anything we taught in the course, instead, they kept demanding more things.

I already started receiving mails from a few people who wants to join in the training in 2020. But, there is no positive answer from my side.

All the course materials are public, the logs are also available. We managed to continue this training for 12 years. This is way more than I could ever imagine.

As I was feeling a bit sad about this, the keynote at Railsconf 2019 from DHH actually helped me a lot to feel better.