PyCoder’s Weekly: Issue #415 (April 7, 2020)(7 hours, 24 minutes ago)

#415 – APRIL 7, 2020
View in Browser »

The PyCoder’s Weekly Logo


Arduino With Python: How to Get Started

Discover how to use Arduino microcontrollers with Python to develop your own electronic projects. You’ll learn how to set up circuits and write applications with the Firmata protocol. You’ll control Arduino inputs and outputs and integrate the board with higher-level apps.
REAL PYTHON course

Things I Wish They Told Me About Multiprocessing in Python [2019]

Python’s multiprocessing module abstracts away a lot of the overhead involved in writing multiprocessing code in Python. But while it might be “easy” to implement multiprocessing in Python, it’s not always easy to do multiprocessing well. In this article, Pamela McA’Nulty shares five tips to help you write better multiprocessing code.
PAMELA MCA'NULTY

Python Developers Are in Demand on Vettery

alt

Vettery is an online hiring marketplace that’s changing the way people hire and get hired. Ready for a bold career move? Make a free profile, name your salary, and connect with hiring managers from top employers today →
VETTERY sponsor

New Sponsorship Program for PyPI

The Packaging Working Group of the Python Software Foundation is launching an all-new sponsorship program to sustain and improve Python’s packaging ecosystem. Funds raised through this program will go directly towards improving the tools that your company uses every day and sustaining the continued operation of the Python Package Index.
PYTHON SOFTWARE FOUNDATION

What the Heck Is pyproject.toml?

“I have seen setuptools users use pyproject.toml because they were ‘told to by ‘ without knowing the entire point behind the file. And so I decided to write this blog post to try and explain to setuptools users why pyproject.toml exists and what it does as it’s the future of packaging in the Python ecosystem.”
BRETT CANNON

Your Configs Suck? Try a Real Programming Language

“In this post, I’ll try to explain why I find most config formats frustrating to use and suggest that using a real programming language (i.e. general purpose one, like Python) is often a feasible and more pleasant alternative for writing configs.”
JESTEM KRÓLICZKIEM

Python Software Foundation Fellow Members for Q1 2020

The PSF Fellow Members for Q1 2020 have been announced. Congratulations to all of the new fellows!
PYTHON SOFTWARE FOUNDATION

Python 2.7.18 Release Candidate 1 Available

“Python 2.7.18 will be the last release of the Python 2.7 series, and thus Python 2.” (It’s not dead, it’s resting!)
PYTHON SOFTWARE FOUNDATION

Discussions

Python Programming Made Me Better at Math

REDDIT

Complete Set of Punctuation Marks for Python?

The string module has a string called punctuation containing several different punctuation characters from the ASCII character set. But not all possible punctuation characters are contained in string.punctuation, such as fancy quotes. What’s the best way to check if any Unicode character is punctuation?
STACK OVERFLOW

How Do You Round a Number to the Next Highest Power of 10?

math.ceil rounds a number to the next highest integer, but how could you round a number to the next highest power of 10 in pure Python?
STACK OVERFLOW

Python Jobs

Python Tutorial Authors Wanted (100% Remote, Freelance)

Real Python

Senior Python API Developer (Remote)

Jefferson Frank

Senior Python Engineer (Remote)

Cybercoders

Sr. Python Developer With Golang (Seattle/Remote)

Empower Professionals

More Python Jobs >>>

Articles & Tutorials

Accelerating Python for Exotic Option Pricing

“In finance, computation efficiency can be directly converted to trading profits sometimes. Quants are facing the challenges of trading off research efficiency with computation efficiency. Using Python can produce succinct research codes, which improves research efficiency. However, vanilla Python code is known to be slow and not suitable for production. In this post, I explore how to use Python GPU libraries to achieve the state-of-the-art performance in the domain of exotic option pricing.”
YI DONG (NVIDIA)

Reworking StringIO Concatenation in Python

Building up a long string by repeatedly concatenating to the end of the “same” string has been considered an anti-Pattern in Python for quite some time. Since strings are immutable, each concatenation should create a new string and therefore incur a performance penalty. Learn how Python 3 optimizes this penalty away, as well as other thoughts on the topic of string concatenation, in this briefing of a recent discussion on the python-ideas mailing list.
JAKE EDGE

Profile, Understand & Optimize Python Code Performance

alt

You can’t improve what you can’t measure. Profile and understand Python code’s behavior and performance (Wall-time, I/O, CPU, HTTP requests, SQL queries). Browse through appealing graphs. Blackfire.io is now available as Public Beta. New features added regularly →
BLACKFIRE.IO sponsor

3 Python Templating Languages You Should (Probably) Never Use

“Python has been around for a long time. In that time, deep in the corners of its system, it has accumulated some almost forgotten templating languages that are well worth poking at. Like cute koalas on top of a eucalyptus tree, happy in their ecological niche, and sometimes as dangerous to work with, these are the templating languages few have heard of—and even fewer should use.”
MOSHE ZADKA

Effective Python and Python at Google Scale

In this episode, Christopher interviews Brett Slatkin about the 2nd edition of his book Effective Python. Brett talks about the revisions he made for the book, and updating it for the newest versions of Python 3. Brett also discusses working on Google App Engine, and what it’s like to develop and maintain Python applications at Google Scale, and working with Guido van Rossum.
REAL PYTHON podcast

Unpacking in Python: Beyond Parallel Assignment

“Unpacking in Python refers to an operation that consists of assigning an iterable of values to a tuple (or list) of variables in a single assignment statement. In this tutorial, [you’ll] learn what iterable unpacking is and how [you] can take advantage of this Python feature to make [your] code more readable, maintainable, and Pythonic.”
LOEDANIS POZO RAMOS

Linked Lists in Python: An Introduction

In this article, you’ll learn what linked lists are and when to use them, such as when you want to implement queues, stacks, or graphs. You’ll also learn how to use collections.deque to improve the performance of your linked lists and how to implement linked lists in your own projects.
REAL PYTHON

When Python Practices Go Wrong

“Just because a programming pattern or convention becomes popular doesn’t always mean that it’s a good idea. This talk digs into the consensus that the Python community has built around what constitutes ‘Pythonic’ code, with a focus on cases where the conventional wisdom may be wrong.”
BRANDON RHODES

The Clean Architecture in Python: How to Write Testable and Flexible Code

“I wrote this blog post because I succeeded in applying the Clean Architecture in two Python projects – both of them reached production and are still being used and developed. This article contains python-specific techniques and tools helpful in embracing the Clean Architecture.”
SEBASTIAN BUCZYŃSKI

Make an Instagram Bot With Python and InstaPy

Learn all about how to use InstaPy to create an Instagram bot that can increase your follower and like count with minimal effort on your end. Along the way, you’ll also learn about two tools that InstaPy uses under the hood: Selenium and the Page Object Pattern.
REAL PYTHON

Python for Middle Schoolers

Stuck in quarantine with a middle schooler? Daniel Lowengrub has put together a fun lesson plan that teaches a beginner how to program in Python with numerous examples and small projects, culminating in an implementation of Tic-Tac-Toe.
DANIEL LOWENGRUB

Thriving in a Remote Developer Environment

“On this episode, I’ll exchange stories about working from home with Jayson Phillips. He’s been writing code and managing a team from his home office for years and has brought a ton of great tips to share with us all.”
TALK PYTHON podcast

Register: Free Intro to Python Workshop for Beginners

Join Metis on April 16 for One Hour at Bootcamp, a free workshop introducing beginner programmers to the fundamentals of Python. Learn live from one of Metis’s top instructors.
METIS sponsor

Understand Django: Templates for User Interfaces

When your Django application sends back a response with your user interface, templates are the tool you’ll use to produce that user interface. This article looks at what templates are and how to use them.
MATT LAYMAN • Shared by Matt Layman

Getting Started With Django Middleware

“Django comes with a lot of useful features. One of them is middleware. In this post I’ll give a short explanation how middleware works and how to start writing your own.”
PAWEŁ FERTYK

Asynchronous Tasks With Django and Celery

This post looks at how to configure Celery to handle long-running tasks in a Django app.
MICHAEL HERMAN • Shared by Michael Herman

Dictionary Merging and Updating in Python 3.9

YONG CUI

Building pyarrow With CUDA Support

RANDY ZWITCH

Projects & Code

matplotlib-3d: Experimental 3D Axis for Matplotlib

GITHUB.COM/ROUGIER

django-anon: Anonymize Your Production Data

GITHUB.COM/TESORIO • Shared by Caio Ariede

check-wheel-contents: Quality Checker for Python Wheels

GITHUB.COM/JWODDER

confight: One Simple Way of Parsing Configs

GITHUB.COM/AVATURE

Simulating Concurrent Writes to Sqlite3 With multiprocessing and pytest

GITHUB.COM/JOEDOUGHERTY

NarrowEscapeSimulator: Python 3 Library for Simulating Brownian Motion

GITHUB.COM/SIRSHARPEST

moderngl: Modern OpenGL Binding for Python

GITHUB.COM/MODERNGL

co2meter: USB CO2 Monitoring With Python

GITHUB.COM/VFILIMONOV

cliche: Build a Simple Command-Line Interface From Your Functions

GITHUB.COM/KOOTENPV

Events

PyCon US 2020 (Remote)

PYCON.ORG

Virtual Nationwide Django Meetup

April 15, 2020
MEETUP.COM


Happy Pythoning!
This was PyCoder’s Weekly Issue #415.
View in Browser »

alt

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

PyCharm: Webinar: “How To Build Real-Time Interactions In Your Django 3 App” with Calvin Hendryx-Parker(12 hours, 49 minutes ago)

Django 3 has been making the rounds, so time for a webinar showing how to use the new features within PyCharm Professional. Calvin Hendryx-Parker from Six Feet Up, previous webinar presenter, is returning to give us the highlights.

  • Wednesday, April 22
  • 12:00 – 13:00 PM EDT
  • Register Now
  • Aimed at developers with basic Django and PyCharm experience

How To Build Real-Time Interactions In Your Django 3 App

Outline

Django 3 brings new features and possibilities. Calvin Hendryx-Parker, previous webinar guest, gives us a tour from a PyCharm perspective:

  • Introduction of new features in Django 3
  • How to use those new features in your app
  • Live coding of async features into an app using PyCharm

Speaking To You

Calvin Hendryx-Parker is the co-founder and CTO of Six Feet Up, a Python web application development company focused on deploying content management systems, intranets and portals, as well as custom web apps using Django, Pyramid and Plone. Under Calvin’s technical leadership, Six Feet Up has served organizations like Amtrak, Eli Lilly, NASA, UCLA and the United Nations.

As an advocate of open source, Calvin is also the founder and organizer of the IndyPy meetup group and Pythology training series in Indianapolis. In 2016 Calvin was nominated for a MIRA Tech Educator of the Year Award.

Real Python: Arduino With Python: How to Get Started(12 hours, 54 minutes ago)

Microcontrollers have been around for a long time, and they’re used in everything from complex machinery to common household appliances. However, working with them has traditionally been reserved for those with formal technical training, such as technicians and electrical engineers.

The emergence of Arduino has made electronic application design much more accessible to all developers. In this course, you’ll discover how to use Arduino with Python to develop your own electronic projects.

You’ll cover the basics of Arduino with Python and learn how to:

  • Set up electronic circuits
  • Set up the Firmata protocol on Arduino
  • Write basic applications for Arduino in Python
  • Control analog and digital inputs and outputs
  • Integrate Arduino sensors and switches with higher-level apps
  • Trigger notifications on your PC and send emails using Arduino

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

S. Lott: The COBOL Problem(18 hours, 54 minutes ago)

Audrey Watters
⁦‪@audreywatters‬⁩
Pet peeve: technologists who sneer at the longevity of programming languages like COBOL. Ain't nobody made it to the moon on react.js. And nobody is willing to replace critical aging software with your new tangled, untested, VC funded framework
4/4/20, 6:24 PM

Download the Twitter app

It's a tweet, so I know there's no room for depth here.

As it is, it's absolutely correct. Allow me to add to it.

First. Replacing COBOL with something shiny and new is more-or-less impossible. Replacing COBOL is a two-step job.

1. Replace the COBOL with something that's nearly identical but written in a new language. Python. Java. Scala. Whatevs. Language doesn't matter. What matters is the hugeness of this leap.

2. Once the COBOL is gone and the mainframe powered off, then you can rebuild things yet again to create RESTful API's and put many shiny things around it.

Second. Replacing COBOL is essential. Software is a form of knowledge capture. If the language (and tools) have become opaque, then the job of knowledge capture has failed. Languages drift. The audience is in a constant state of flux. New translations are required.

Let's talk about the "Nearly Identical But In A New Language."

Nearly Identical

COBOL code has two large issues in general
  • Data. The file layouts are very hard to work with. I know a lot about this. 
  • Processing. The code has crap implementations of common data structures. I know. I wrote some. There's more, we'll get to it.
We have -- for the most part -- two kinds of COBOL code in common use.
  • Batch processing. Once upon a time, we called it "Programming in the Large." The Z/OS Job Control Language (JCL) was a kind of shell script or AWS Step Function state transition map among applications. This isn't easy to deal with because the overall data flow is not a simple Directed Acyclic Graph (DAG.) It has cycles and state changes.
  • Interactive (once called "on-line") processing. We called it OLTP: On-Line Transaction Processing. There are two common frameworks, CICS and IMS, and both are complicated.
Okay. Big Breath. What do we *DO*?

Here's the free consulting part.

You have to run the new and old side-by-side until you're sick of the errors and poor performance of the old machine.

You have to migrate incrementally, one app at a time.

It's hellishly expensive to positively determine what the COBOL really did. You can't easily do a "clean-room" conversion by writing intermediate specifications. You must read the COBOL and rewrite it into Python (or Java or Scala or whatever.)

You cannot unit test your way to success here, because you never really knew what the COBOL does/did. All you can do is extract example records and use those to build Gherkin-language acceptance tests using a template like this. GIVEN a source document WHEN the app runs THEN the output document matches the example. 

In effect, you're going to do TDD on the COBOL, replacing COBOL with Python essentially 1-for-1 until you have a test suite that passes.

Don't do this alphabetically, BTW. 

The processing graph for COBOL will include three essential design patterns for programs. "Edit" programs validate and possibly merge input files. "Update" programs will apply changes to master files or databases. "Report" programs will produce useful reports and feeds for reporting systems that involve yet more data derivation and merging.

  1. Find the updates. Convert them first. They will involve the most knowledge capture, A/K/A "Business Logic."  There will be a lot of special cases and exceptions. You will find latent bugs that have always been there.
  2. Convert the programs that produce files for the updates, working forward in the graph.
  3. The "reporting" is generally a proper DAG, and should be easier to deal with than the updates and edits. You never know, but the reporting apps are filled with redundancy. Tons of reporting programs are minor variations on each other, often built as copy-pasta from some original text and then patched haphazardly. Most of them can be replaced with a tool to emit CSV files as an interim step.
Each converted application requires two new steps injected into the COBOL batch jobs.
  • Before an update runs, the files are pushed to some place where they can be downloaded.
  • The app runs as it always had. For now.
  • After the update, the results are pushed, also.
This changes merely slow things down with file transfers. It provides fodder for parallel testing.

Then. 

Two changes are made so the job now looks like this.
  • Before an update runs, the files are pushed to some place where they can be downloaded. (No change here.)
  • Kill time polling the file location, waiting for the file to be created externally. (The old app is still around. We could run it if we wanted to.) 
  • After the update, download the results from the external location.
This file-copy-and-parallel-run dance can, of course, be optimized if you take whole streams of edit-update processing and convert them as a whole.

Yes, But, The COBOL Is Complicated

No. It's not.

It's a lot of code working around language limitations. There aren't many design patterns, and they're easy to find.
  1. Read, Validate, Write. The validation is quirky, but generally pretty easy to understand. In the long run, the whole thing is a JSONSchema document. But for now, there may be some data cleansing or transformation steps buried in here.
  2. Merged Reading. Execute the Transaction. Write. The transaction execution updates are super important. These are the state changes in object classes. They're often entangled among bad representations of data. 
  3. Cached Data. A common performance tweak is to read reference data ("Lookups") into an array. This was often hellishly complex because... well... COBOL. It was a Python dict, for the love of God, there's nothing to it. Now. Then. Well. It was tricky.
  4. Accumulators. Running totals and counts were essential for audit purposes. The updates could be hidden anywhere. Anywhere. Not part of the overall purpose, but necessary anyway.
  5. Parameter Processing. This can be quirky. Some applications had a standard dataset with parameters like the as-of-date for the processing. Some applications prompted an operator. Some had other quirky ways of handling the parameters.
The bulk of the code isn't very complex. It's quirky. But not complicated.

The absolute worst applications were summary reports with a hierarchy. We called these "control break" reports. I don't know why. Each level of the hierarchy had its own accumulators. The data had to be properly sorted. It was complicated. 

Do Not Convert these. Find any data cleansing or transformation and simply pour the data into a CSV file and let the users put it into a spreadsheet.

Right now. We have to keep the lights on. COBOL apps have to be kept operational to manage unemployment benefits through the pandemic.

But once we're out of this. We need to get rid of the COBOL.

And we need to recognize that all code expires and we need to plan for expiration. 

Codementor: Flask Delicious Tutorial : Building a Library Management System Part 2 - Start With A Loaded Skeleton(19 hours, 18 minutes ago)

In this part, we include some front-end libs to have a great start!

Mike Driscoll: Python 101 – Working with Strings(21 hours, 47 minutes ago)

You will be using strings very often when you program. A string is a series of letters surrounded by single, double or triple quotes. Python 3 defines string as a “Text Sequence Type”. You can cast other types to a string using the built-in str() function.

In this article you will learn how to:

  • Create strings
  • String methods
  • String formatting
  • String concatenation
  • String slicing

Let’s get started by learning the different ways to create strings!

Creating Strings

Here are some examples of creating strings:

name = 'Mike'
first_name = 'Mike'
last_name = "Driscoll"
triple = """multi-line
string"""

When you use triple quotes, you may use three double quotes at the beginning and end of the string or three single quotes. Also, note that using triple quotes allows you to create multi-line strings. Any whitespace within the string will also be included.

Here is an example of converting an integer to a string:

>>> number = 5
>>> str(number)
'5'

In Python, backslashes can be used to create escape sequences. Here are a couple of examples:

  • \b – backspace
  • \n – line feed
  • \r – ASCII carriage return
  • \t – tab

There are several others that you can learn about if you read Python’s documentation.

You can also use backslashes to escape quotes:

>>> 'This string has a single quote, \', in the middle'
"This string has a single quote, ', in the middle"

If you did not have the backslash in the code above, you would receive a SyntaxError:

>>> 'This string has a single quote, ', in the middle'
Traceback (most recent call last):
  Python Shell, prompt 59, line 1
invalid syntax: <string>, line 1, pos 38

This occurs because the string ends at that second single quote. It is usually better to mix double and single quotes to get around this issue:

>>> "This string has a single quote, ', in the middle"
"This string has a single quote, ', in the middle"

In this case, you create the string using double quotes and put a single quote inside of it. This is especially helpful when working with contractions, such as “don’t”, “can’t”, etc.

Now let’s move along and see what methods you can use with strings!

String Methods

In Python, everything is an object. You will learn how useful this can be in chapter 18 when you learn about introspection. For now, just know that strings have methods (or functions) that you can call on them.

Here are three examples:

>>> name = 'mike'
>>> name.capitalize()
'Mike'
>>> name.upper()
'MIKE'
>>> 'MIke'.lower()
'mike'

The method names give you a clue as to what they do. For example, .capitalize() will change the first letter in the string to a capital letter.

To get a full listing of the methods and attributes that you can access, you can use Python’s built-in dir() function:

>>> dir(name)
['__add__', '__class__', '__contains__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__',
'__getattribute__', '__getitem__', '__getnewargs__', '__gt__', '__hash__', '__init__', '__init_subclass__',
'__iter__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__',
'__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'capitalize',
'casefold', 'center', 'count', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'format_map', 'index',
'isalnum', 'isalpha', 'isascii', 'isdecimal', 'isdigit', 'isidentifier', 'islower', 'isnumeric', 'isprintable',
'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'maketrans', 'partition', 'replace',
'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip',
'swapcase', 'title', 'translate', 'upper', 'zfill']

The first third of the listing are special methods that are sometimes called “dunder methods” (AKA double-underscore methods) or “magic methods”. You can ignore these for now as they are used more for intermediate and advanced use-cases. The items in the list above that don’t have double-underscores at the beginning are the ones that you will probably use the most.

You will find that the .strip() and .split() methods are especially useful when parsing or manipulating text.

You can use .strip() and its variants, .rstrip() and .lstrip() to strip off white space from the string, including tab and new line characters. This is especially useful when you are reading in a text file that you need to parse.

In fact, you will often end up stripping end-of-line characters from strings and then using .split() on the result to parse out sub-strings.

Let’s do a little exercise where you will learn how to parse out the 2nd word in a string.

To start, here’s a string:

>>> my_string = 'This is a string of words'
'This is a string of words'

Now to get the parts of a string, you can call .split(), like this:

>>> my_string.split()
['This', 'is', 'a', 'string', 'of', 'words']

The result is a list of strings. Now normally you would assign this result to a variable, but for demonstration purposes, you can skip that part.

Instead, since you now know that the result is a string, you can use list slicing to get the second element:

>>> 'This is a string of words'.split()[1]
'is'

Remember, in Python, lists elements start at 0 (zero), so when you tell it you want element 1 (one), that is the second element in the list.

When doing string parsing for work, I personally have found that you can use the .strip() and .split() methods pretty effectively to get almost any data that you need. Occasionally you will find that you might also need to use Regular Expressions (regex), but most of the time these two methods are enough.

String Formatting

String formatting or string substitution is where you have a string that you would like to insert into another string. This is especially useful when you need to do a template, like a form letter. But you will use string substitution a lot for debugging output, printing to standard out and much more.

Python has three different ways to accomplish string formatting:

  • Using the % Method
  • Using .format()
  • Using formatted string literals (f-strings)

This book will focus on f-strings the most and also use .format() from time-to-time. But it is good to understand how all three work.

Let’s take a few moments to learn more about string formatting.

Formatting Strings Using %s (printf-style)

Using the % method is Python’s oldest method of string formatting. It is sometimes referred to as “printf-style string formatting”. If you have used C or C++ in the past, then you may already be familiar with this type of string substitution. For brevity, you will learn the basics of using % here.

Note: This type of formatting can be quirky to work with and has been known to lead to common errors such as failing to display Python tuples and dictionaries incorrectly. Using either of the other two methods is preferred in that case.

The most common use of using the % sign is when you would use %s, which means convert any Python object to a string using str().

Here is an example:

>>> name = 'Mike'
>>> print('My name is %s' % name)
My name is Mike

In this code, you take the variable name and insert it into another string using the special %s syntax. To make it work, you need to use % outside of the string followed by the string or variable that you want to insert.

Here is a second example that shows that you can pass in an int into a string and have it automatically converted for you:

>>> age = 18
>>> print('You must be at least %s to continue' % age)
You must be at least 18 to continue

This sort of thing is especially useful when you need to convert an object but don’t know what type it is.

You can also do string formatting with multiple variables. In fact, there are two ways to do this.

Here’s the first one:

>>> name = 'Mike'
>>> age = 18
>>> print('Hello %s. You must be at least %i to continue!' % (name, age))
Hello Mike. You must be at least 18 to continue!

In this example, you create two variables and use %s and %i. The %i indicates that you are going to pass an integer. To pass in multiple items, you use the percent sign followed by a tuple of the items to insert.

You can make this clearer by using names, like this:

>>> print('Hello %(name)s. You must be at least %(age)i to continue!' % {'name': name, 'age': age})
Hello Mike. You must be at least 18 to continue!

When the argument on the right side of the % sign is a dictionary (or another mapping type), then the formats in the string must refer to the parenthesized key in the dictionary. In other words, if you see %(name)s, then the dictionary to the right of the % must have a name key.

If you do not include all the keys that are required, you will receive an error:

>>> print('Hello %(name)s. You must be at least %(age)i to continue!' % {'age': age})
Traceback (most recent call last):
   Python Shell, prompt 23, line 1
KeyError: 'name'

For more information about using the printf-style string formatting, you should see the following link:

https://docs.python.org/3/library/stdtypes.html#printf-style-string-formatting

Now let’s move on to using the .format() method.

Formatting Strings Using .format()

Python strings have supported the .format() method for a long time. While this book will focus on using f-strings, you will find that .format() is still quite popular.

For full details on how formatting works, see the following:

https://docs.python.org/3/library/string.html#formatstrings

Let’s take a look at a few short examples to see how .format() works:

>>> age = 18
>>> name = 'Mike'
>>> print('Hello {}. You must be at least {} to continue!'.format(name, age))
Hello Mike. You must be at least 18 to continue!

This example uses positional arguments. Python looks for two instances of {} and will insert the variables accordingly. If you do not pass in enough arguments, you will receive an error like this:

>>> print('Hello {}. You must be at least {} to continue!'.format(age))
Traceback (most recent call last):
    Python Shell, prompt 33, line 1
IndexError: tuple index out of range

This error indicates that you do not have enough items inside the .format() call.

You can also use named arguments in a similar way to the previous section:

>>> age = 18
>>> name = 'Mike'
>>> print('Hello {name}. You must be at least {age} to continue!'.format(name=name, age=age))
Hello Mike. You must be at least 18 to continue!

Instead of passing a dictionary to .format(), you can pass in the parameters by name. In fact, if you do try to pass in a dictionary, you will receive an error:

>>> print('Hello {name}. You must be at least {age} to continue!'.format({'name': name, 'age': age}))
Traceback (most recent call last):
  Python Shell, prompt 34, line 1
KeyError: 'name'

There is a workaround for this though:

>>> print('Hello {name}. You must be at least {age} to continue!'.format(**{'name': name, 'age': age}))
Hello Mike. You must be at least 18 to continue!

This looks a bit weird, but in Python when you see a double asterisk (**) used like this, it means that you are passing named parameters to the function. So Python is converting the dictionary to name=name, age=age for you.

You can also use repeat a variable multiple times in the string using .format():

>>> name = 'Mike'
>>> print('Hello {name}. Why do they call you {name}?'.format(name=name))
Hello Mike. Why do they call you Mike?

Here you refer to {name} twice in the string and you are able to replace both of them using .format().

If you want, you can also interpolate values using numbers:

>>> print('Hello {1}. You must be at least {0} to continue!'.format(name, age))
Hello 18. You must be at least Mike to continue!

Because most things in Python start at 0 (zero), in this example you ended up passing the age to {1} and the name to {0}.

A common coding style when working with .format() is to create a formatted string and save it to a variable to be used later:

>>> age = 18
>>> name = 'Mike'
>>> greetings = 'Hello {name}. You must be at least {age} to continue!'
>>> greetings.format(name=name, age=age)
'Hello Mike. You must be at least 18 to continue!'

This allows you to reuse greetings and pass in updated values for name and age later on in your program.

You can also specify the string width and alignment:

>>> '{:<20}'.format('left aligned')
'left aligned        '
>>> '{:>20}'.format('right aligned')
'       right aligned'
>>> '{:^20}'.format('centered')
'      centered      '

Left aligned is the default. The colon (:) tells Python that you are going to apply some kind of formatting. In the first example, you are specifying that the string be left aligned and 20 characters wide. The second example is also 20 characters wide, but it is right aligned. Finally the ^ tells Python to center the string within the 20 character string.

If you want to pass in a variable like in the previous examples, here is how you would do that:

>>> '{name:^20}'.format(name='centered')
'      centered      '

Note that the name must come before the : inside of the {}.

At this point, you should be pretty familiar with the way .format() works.

Let’s go ahead and move along to f-strings!

Formatting Strings with f-strings

Formatted string literals or f-strings are strings that have an “f” at the beginning and curly braces inside of them that contain expressions, much like the ones you saw in the previous section. These expressions tell the f-string about any special processing that needs to be done to the inserted string, such as justification, float precision, etc.

The f-string was added in Python 3.6. You can read more about it and how it works by checking out PEP 498 here:

https://www.python.org/dev/peps/pep-0498/

The expressions that are contained inside of f-strings are evaluated at runtime. This makes it impossible to use an f-string as a docstring to a function, method or class if it contains an expression. The reason being that docstrings are defined at function definition time.

Let’s go ahead and look at a simple example:

>>> name = 'Mike'
>>> age = 20
>>> f'Hello {name}. You are {age} years old'
'Hello Mike. You are 20 years old'

Here you create the f-string by putting an “f” right before the single, double or triple quote that begins your string. Then inside of the string, you use the curly braces, {}, to insert variables into your string.

However, your curly braces must enclose something. If you create an f-string with empty braces, you will get an error:

>>> f'Hello {}. You are {} years old'
SyntaxError: f-string: empty expression not allowed

The f-string can do things that neither %s nor .format() can do though. Because of the fact that f-strings are evaluated at runtime, you can put any valid Python expression inside of them.

For example, you could increase the age variable:

>>> age = 20
>>> f'{age+2}'
'22'

Or call a method or function:

>>> name = 'Mike'
>>> f'{name.lower()}'
'mike'

You can also access dictionary values directly inside of an f-string:

>>> sample_dict = {'name': 'Tom', 'age': 40}
>>> f'Hello {sample_dict["name"]}. You are {sample_dict["age"]} years old'
'Hello Tom. You are 40 years old'

However, backslashes are not allowed in f-string expressions:

>>> print(f'My name is {name\n}')
SyntaxError: f-string expression part cannot include a backslash

But you can use backslashes outside of the expression in an f-string:

>>> name = 'Mike'
>>> print(f'My name is {name}\n')
My name is Mike

One other thing that you can’t do is add a comment inside of an expression in an f-string:

>>> f'My name is {name # name of person}'
SyntaxError: f-string expression part cannot include '#'

In Python 3.8, f-strings added support for =, which will expand the text of the expression to include the text of the expression plus the equal sign and then the evaluated expression. That sounds kind of complicated, so let’s look at an example:

>>> username = 'jdoe'
>>> f'Your {username=}'
"Your username='jdoe'"

This example demonstrates that the text inside of the expression, username= is added to the output followed by the actual value of username in quotes.

f-strings are very powerful and extremely useful. They will simplify your code quite a bit if you use them wisely. You should definitely give them a try.

Let’s find out what else you can do with strings!

String Concatenation

Strings also allow concatenation, which is a fancy word for joining two strings into one.

To concatenate strings together, you can use the + sign:

>>> first_string = 'My name is'
>>> second_string = 'Mike'
>>> first_string + second_string
'My name isMike'

Oops! It looks like the strings merged in a weird way because you forgot to add a space to the end of the first_string. You can change it like this:

>>> first_string = 'My name is '
>>> second_string = 'Mike'
>>> first_string + second_string
'My name is Mike'

Another way to merge strings is to use the .join() method. The .join() method accepts an iterable, such as a list, of strings and joins them together.

>>> first_string = 'My name is '
>>> second_string = 'Mike'
>>> ''.join([first_string, second_string])
'My name is Mike'

This will make the strings join right next to each other. You could put something inside of the string that you are joining though:

>>> '***'.join([first_string, second_string])
'My name is ***Mike'

In this case, it will join the first string to *** plus the second string.

More often than not, you can use an f-string rather than concatenation or .join() and the code will be easier to follow.

String Slicing

Slicing in strings works in much the same way that it does for Python lists. Let’s take the string “Mike”. The letter “M” is at position zero and the letter “e” is at position 3.

If you want to grab characters 0-3, you would use this syntax: my_string[0:4]

What that means is that you want the substring starting at position zero up to but not including position 4.

Here are a few examples:

>>> 'this is a string'[0:4]
'this'
>>> 'this is a string'[:4]
'this'
>>> 'this is a string'[-4:]
'ring'

The first example grabs the first four letters from the string and returns them. If you want to, you can drop the zero as that is the default and use [:4] instead, which is what example two does.

You can also use negative position values. So [-4:] means that you want to start at the end of the string and get the last four letters of the string.

You should play around with slicing on your own and see what other slices you can come up with.

Wrapping Up

Python strings are powerful and quite useful. They can be created using single, double or triple quotes. Strings are objects, so they have methods. You also learned about string concatenation, string slicing and three different methods of string formatting.

The newest flavor of string formatting is the f-string. It is also the most powerful and the currently preferred method for formatting strings.

Related Reading

The post Python 101 – Working with Strings appeared first on The Mouse Vs. The Python.

Podcast.__init__: Building The Seq Language For Bioinformatics(1 day, 1 hour ago)

Bioinformatics is a complex and computationally demanding domain. The intuitive syntax of Python and extensive set of libraries make it a great language for bioinformatics projects, but it is hampered by the need for computational efficiency. Ariya Shajii created the Seq language to bridge the divide between the performance of languages like C and C++ and the ecosystem of Python with built-in support for commonly used genomics algorithms. In this episode he describes his motivation for creating a new language, how it is implemented, and how it is being used in the life sciences. If you are interested in experimenting with sequencing data then give this a listen and then give Seq a try!

Summary

Bioinformatics is a complex and computationally demanding domain. The intuitive syntax of Python and extensive set of libraries make it a great language for bioinformatics projects, but it is hampered by the need for computational efficiency. Ariya Shajii created the Seq language to bridge the divide between the performance of languages like C and C++ and the ecosystem of Python with built-in support for commonly used genomics algorithms. In this episode he describes his motivation for creating a new language, how it is implemented, and how it is being used in the life sciences. If you are interested in experimenting with sequencing data then give this a listen and then give Seq a try!

Announcements

  • Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
  • When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With 200 Gbit/s private networking, node balancers, a 40 Gbit/s public network, fast object storage, and a brand new managed Kubernetes platform, all controlled by a convenient API you’ve got everything you need to scale up. And for your tasks that need fast computation, such as training machine learning models, they’ve got dedicated CPU and GPU instances. Go to pythonpodcast.com/linode to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
  • You listen to this show to learn and stay up to date with the ways that Python is being used, including the latest in machine learning and data analysis. For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on great conferences. And now, the events are coming to you, with no travel necessary! We have partnered with organizations such as ODSC, and Data Council. Upcoming events include the Observe 20/20 virtual conference on April 6th and ODSC East which has also gone virtual starting April 16th. Go to pythonpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today.
  • Your host as usual is Tobias Macey and today I’m interviewing Ariya Shajii about Seq, a programming language built for bioinformatics and inspired by Python

Interview

  • Introductions
  • How did you get introduced to Python?
  • Can you start by describing what Seq is and your motivation for creating it?
    • What was lacking in other languages or libraries for your use case that is made easier by creating a custom language?
    • If someone is already working in Python, possibly using BioPython, what might motivate them to consider migrating their work to Seq?
  • Can you give an impression of the scope and nature of the tasks or projects that a biologist or geneticist might build with Seq?
  • What was your process for identifying and prioritizing features and algorithms that would be beneficial to the target audience?
  • For someone using Seq can you describe their workflow and how it might differ from performing the same task in Python?
  • How is Seq implemented?
    • What are some of the features that are included to simplify the work of bioinformatics?
    • What was your process of designing the language and runtime?
    • How has the scope or direction of the project evolved since it was first conceived?
  • What impact do you anticipate Seq having on the domain of bioinformatics and genomics?
  • What have you found to be the most interesting, unexpected, and/or challenging aspects of building a language for this problem domain?
  • What is in store for the future of Seq?

Keep In Touch

Picks

Closing Announcements

  • Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management.
  • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
  • If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@podcastinit.com) with your story.
  • To help other people find the show please leave a review on iTunes and tell your friends and co-workers
  • Join the community in the new Zulip chat workspace at pythonpodcast.com/chat

Links

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

Wing Tips: Debug Python Services Running on AWS with Wing Pro(1 day, 1 hour ago)

In this Wing Tip we're continuing our look at how to use Wing Pro to remotely develop Python code running on an AWS instance. This time we'll set up remote debugging of code that is launched from outside of the IDE. This can be useful in debugging web apps and other services.

Prerequisites

Before getting started, you will need an AWS instance that you can SSH into, and you'll want to set up a Wing project for that AWS instance. If you don't already have this working, take a look at last's weeks Wing Tip: Remote Python Development on AWS with Wing Pro.

The small debugging example given in the above link launched the remote code being debugged from the IDE. The code actually runs on the AWS instance, but Wing launches it with the help of its remote agent. Now we're instead going to set up debugging remote code that is launched outside of control of the IDE.

This is done by (1) using Wing Pro's remote development support to set up a reverse SSH tunnel to the remote AWS instance, so that the debugger can connect back to the IDE, and then (2) inserting an import into code that starts debug and makes the connection to the IDE.

Setting up Remote Listening

In the project you have set up for your remote AWS instance, click on the bug icon in the lower left of Wing's window to check on Accept Debug Connections:

/images/blog/aws-2/debug-menu.png

Now if you hover the mouse over the bug icon, you will see that Wing reports that it is listening for debug connections on the remote host on port 50050, as well as listening on the local host:

/images/blog/aws-2/debug-status.png

Starting Debug

To keep things simple, rather than setting up a web development framework or other service, we'll test with a simple helloworld.py example that we launch manually from the command line outside of Wing.

You can create this file on the AWS instance by right-clicking on a directory you previously added to Wing's Project tool and selecting Create New File. This collects the new file name in a dialog or, in some keyboard configurations, in the status area at the bottom of Wing's window, and then opens the file in the editor.

Then paste in the following code and save the file:

import wingdbstub
x = 1
print("Hello", x)
x +=1
print("Goodbye",x)

Notice that the first line import wingdbstub starts debug and connects to the IDE through the reverse SSH tunnel you set up in the previous section.

In order for this to work, you need to copy the automatically preconfigured file ~/.wingpro7/remote-#.#.#.#/wingdbstub.py on the remote host (where #.#.#.# is the version of Wing you running) into the same directory as helloworld.py. This can be done by opening the file in Wing and using Save As. Or just connect to your AWS instance, cd into your target directory, and copy it into place:

cp ~/.wingpro7/remote-#.#.#.#/wingdbstub.py .

If you are using Lightsail, a convenient way to connect to your instance is Connect Using SSH under the Connect tab in the Amazon Lightsail management interface.

Finally, set a breakpoint on line 3 of the test code so it will not just run to completion, but instead will stop in Wing's debugger. This is done by clicking on the leftmost margin in the editor:

/images/blog/aws-2/breakpoint.png

Now you can start debug just by launching the test code outside of Wing:

python3 helloworld.py

The first time you do this, Wing may refuse the connection and ask whether you want to accept the security token for this host. This happens if you installed the remote agent earlier from a different Wing installation:

/images/blog/aws-2/security-token.png

If this dialog appears, click Accept and then start the test code a second time:

python3 helloworld.py

This time Wing will accept the connection and should stop on your breakpoint:

/images/blog/aws-2/stopped.png

From here, you can inspect your debug process in the Debug Console or Stack Data, set other breakpoints or conditional breakpoints, step through code, and so forth. For more information on the capabilities of Wing's debugger, see the Debugger Quick Start or the tutorial in Wing's Help menu.

Notes

To actually set up Wing to work with a web development framework on your AWS instance, see our How-Tos for Flask, Django, or other web development frameworks. These explain additional configuration that may be needed in each case.

Wing's debugger also provides a simple debugger API that may be useful, for example in developing a way to turn the debugger on and off on demand or controlling which threads are debugged.

For detailed documentation see the Debugger and Advanced Debugging Topics chapters in Wing Pro's user manual.



That's it for now! We'll be back soon with more Wing Tips for Wing Python IDE.

As always, please don't hesitate to email support@wingware.com if you run into problems or have any questions.

Go Deh: Spin the table: Solution!(1 day, 10 hours ago)

An answer to a puzzle set by Matt Parker on his YouTube channel:



The Puzzle:


  1. A Circular table with positions 1..7
  2. Delegates numbered 1..7 arranged arbitrarily around the table.
  3. Delegates start with only one match to a correct seating position.
  4. Is it ALWAYS possible to spin the table and get two or more matches?
Find arrangements where it is not possible to spin the table to get more than one match.

The Model.

Let's number the positions on the table with integers 1 .. 7; and also give the delegates numbers 1 .. 7. If the delegate number equals the table number then that constitutes a seating match.

Storing the table order as a fixed list of [1, 2, ..7], the all the rotations of the table  can be modelled by the additional six rotations of that initial list found by popping the last number from the last rotation and appending to the front of the list, i.e:
[[1, 2, .. 7],
 [7, 1, .. 6],
 [6, 7, 1, .. 5],
 ..
 [2, 3, .. 7, 1]]

This is stored in variable spun.

The permutations of delegate numbers 1 .. 7 are all the possible orderings of delegates.
Every "base" perm will have six other perms which are just other rotations of the "base" ordering . If only the permutations generated with an initial 1 are used, (for example), then we can assume that any of the other six rotations of the permutation will also be an answer.

The Code.



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# -*- coding: utf-8 -*-
"""
Spin the table.

Created on Wed Mar 25 19:59:49 2020

@author: Paddy3118
"""

from itertools import permutations


table = list(range(1, 8))
spun = [] # spinning table positions
for i in range(7):
spun.append(table)
table = table[-1:] + table[:-1] # rotate once

def match_count(table, delegates):
return sum(1 for t, d in zip(table, delegates)
if t == d)

def max_matches(delegates):
return max(match_count(table, delegates) for table in spun)

def spin_the_table():
return [d for d in permutations(table)
if d[0] == 1 and match_count(table, d) == 1 and max_matches(d) < 2]


if __name__ == '__main__':
answer = spin_the_table()
print(f"Found {len(answer)} initial arrangements of the delegates:")
for a in answer:
print(' ', str(a)[1:-1])
print("The 6 rotations of each individual answer above are also solutions")

Answer:



Found 19 initial arrangements of the delegates:
1, 3, 5, 7, 2, 4, 6
1, 3, 6, 2, 7, 5, 4
1, 3, 7, 6, 4, 2, 5
1, 4, 2, 7, 6, 3, 5
1, 4, 6, 3, 2, 7, 5
1, 4, 7, 2, 6, 5, 3
1, 4, 7, 3, 6, 2, 5
1, 4, 7, 5, 3, 2, 6
1, 5, 2, 6, 3, 7, 4
1, 5, 4, 2, 7, 3, 6
1, 5, 7, 3, 6, 4, 2
1, 6, 2, 5, 7, 4, 3
1, 6, 4, 2, 7, 5, 3
1, 6, 4, 3, 7, 2, 5
1, 6, 4, 7, 3, 5, 2
1, 6, 5, 2, 4, 7, 3
1, 7, 4, 6, 2, 5, 3
1, 7, 5, 3, 6, 2, 4
1, 7, 6, 5, 4, 3, 2
The 6 rotations of each individual answer above are also solutions


Matt Parker's Solution Video: (He mentions my solution at time 05:00).

END.

Real Python: How to Make an Instagram Bot With Python and InstaPy(1 day, 12 hours ago)

What do SocialCaptain, Kicksta, Instavast, and many other companies have in common? They all help you reach a greater audience, gain more followers, and get more likes on Instagram while you hardly lift a finger. They do it all through automation, and people pay them a good deal of money for it. But you can do the same thing—for free—using InstaPy!

In this tutorial, you’ll learn how to build a bot with Python and InstaPy, which automates your Instagram activities so that you gain more followers and likes with minimal manual input. Along the way, you’ll learn about browser automation with Selenium and the Page Object Pattern, which together serve as the basis for InstaPy.

In this tutorial, you’ll learn:

  • How Instagram bots work
  • How to automate a browser with Selenium
  • How to use the Page Object Pattern for better readability and testability
  • How to build an Instagram bot with InstaPy

You’ll begin by learning how Instagram bots work before you build one.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

How Instagram Bots Work

How can an automation script gain you more followers and likes? Before answering this question, think about how an actual person gains more followers and likes.

They do it by being consistently active on the platform. They post often, follow other people, and like and leave comments on other people’s posts. Bots work exactly the same way: They follow, like, and comment on a consistent basis according to the criteria you set.

The better the criteria you set, the better your results will be. You want to make sure you’re targeting the right groups because the people your bot interacts with on Instagram will be more likely to interact with your content.

For example, if you’re selling women’s clothing on Instagram, then you can instruct your bot to like, comment on, and follow mostly women or profiles whose posts include hashtags such as #beauty, #fashion, or #clothes. This makes it more likely that your target audience will notice your profile, follow you back, and start interacting with your posts.

How does it work on the technical side, though? You can’t use the Instagram Developer API since it is fairly limited for this purpose. Enter browser automation. It works in the following way:

  1. You serve it your credentials.
  2. You set the criteria for who to follow, what comments to leave, and which type of posts to like.
  3. Your bot opens a browser, types in https://instagram.com on the address bar, logs in with your credentials, and starts doing the things you instructed it to do.

Next, you’ll build the initial version of your Instagram bot, which will automatically log in to your profile. Note that you won’t use InstaPy just yet.

How to Automate a Browser

For this version of your Instagram bot, you’ll be using Selenium, which is the tool that InstaPy uses under the hood.

First, install Selenium. During installation, make sure you also install the Firefox WebDriver since the latest version of InstaPy dropped support for Chrome. This also means that you need the Firefox browser installed on your computer.

Now, create a Python file and write the following code in it:

 1 from time import sleep
 2 from selenium import webdriver
 3 
 4 browser = webdriver.Firefox()
 5 
 6 browser.get('https://www.instagram.com/')
 7 
 8 sleep(5)
 9 
10 browser.close()

Run the code and you’ll see that a Firefox browser opens and directs you to the Instagram login page. Here’s a line-by-line breakdown of the code:

  • Lines 1 and 2 import sleep and webdriver.
  • Line 4 initializes the Firefox driver and sets it to browser.
  • Line 6 types https://www.instagram.com/ on the address bar and hits Enter.
  • Line 8 waits for five seconds so you can see the result. Otherwise, it would close the browser instantly.
  • Line 10 closes the browser.

This is the Selenium version of Hello, World. Now you’re ready to add the code that logs in to your Instagram profile. But first, think about how you would log in to your profile manually. You would do the following:

  1. Go to https://www.instagram.com/.
  2. Click the login link.
  3. Enter your credentials.
  4. Hit the login button.

The first step is already done by the code above. Now change it so that it clicks on the login link on the Instagram home page:

 1 from time import sleep
 2 from selenium import webdriver
 3 
 4 browser = webdriver.Firefox()
 5 browser.implicitly_wait(5)
 6 
 7 browser.get('https://www.instagram.com/')
 8 
 9 login_link = browser.find_element_by_xpath("//a[text()='Log in']")
10 login_link.click()
11 
12 sleep(5)
13 
14 browser.close()

Note the highlighted lines:

  • Line 5 sets five seconds of waiting time. If Selenium can’t find an element, then it waits for five seconds to allow everything to load and tries again.
  • Line 9 finds the element <a> whose text is equal to Log in. It does this using XPath, but there are a few other methods you could use.
  • Line 10 clicks on the found element <a> for the login link.

Run the script and you’ll see your script in action. It will open the browser, go to Instagram, and click on the login link to go to the login page.

On the login page, there are three important elements:

  1. The username input
  2. The password input
  3. The login button

Next, change the script so that it finds those elements, enters your credentials, and clicks on the login button:

 1 from time import sleep
 2 from selenium import webdriver
 3 
 4 browser = webdriver.Firefox()
 5 browser.implicitly_wait(5)
 6 
 7 browser.get('https://www.instagram.com/')
 8 
 9 login_link = browser.find_element_by_xpath("//a[text()='Log in']")
10 login_link.click()
11 
12 sleep(2)
13 
14 username_input = browser.find_element_by_css_selector("input[name='username']")
15 password_input = browser.find_element_by_css_selector("input[name='password']")
16 
17 username_input.send_keys("<your username>")
18 password_input.send_keys("<your password>")
19 
20 login_button = browser.find_element_by_xpath("//button[@type='submit']")
21 login_button.click()
22 
23 sleep(5)
24 
25 browser.close()

Here’s a breakdown of the changes:

  1. Line 12 sleeps for two seconds to allow the page to load.
  2. Lines 14 and 15 find username and password inputs by CSS. You could use any other method that you prefer.
  3. Lines 17 and 18 type your username and password in their respective inputs. Don’t forget to fill in <your username> and <your password>!
  4. Line 20 finds the login button by XPath.
  5. Line 21 clicks on the login button.

Run the script and you’ll be automatically logged in to to your Instagram profile.

You’re off to a good start with your Instagram bot. If you were to continue writing this script, then the rest would look very similar. You would find the posts that you like by scrolling down your feed, find the like button by CSS, click on it, find the comments section, leave a comment, and continue.

The good news is that all of those steps can be handled by InstaPy. But before you jump into using Instapy, there is one other thing that you should know about to better understand how InstaPy works: the Page Object Pattern.

How to Use the Page Object Pattern

Now that you’ve written the login code, how would you write a test for it? It would look something like the following:

def test_login_page(browser):
    browser.get('https://www.instagram.com/accounts/login/')
    username_input = browser.find_element_by_css_selector("input[name='username']")
    password_input = browser.find_element_by_css_selector("input[name='password']")
    username_input.send_keys("<your username>")
    password_input.send_keys("<your password>")
    login_button = browser.find_element_by_xpath("//button[@type='submit']")
    login_button.click()

    errors = browser.find_elements_by_css_selector('#error_message')
    assert len(errors) == 0

Can you see what’s wrong with this code? It doesn’t follow the DRY principle. That is, the code is duplicated in both the application and the test code.

Duplicating code is especially bad in this context because Selenium code is dependent on UI elements, and UI elements tend to change. When they do change, you want to update your code in one place. That’s where the Page Object Pattern comes in.

With this pattern, you create page object classes for the most important pages or fragments that provide interfaces that are straightforward to program to and that hide the underlying widgetry in the window. With this in mind, you can rewrite the code above and create a HomePage class and a LoginPage class:

from time import sleep

class LoginPage:
    def __init__(self, browser):
        self.browser = browser

    def login(self, username, password):
        username_input = self.browser.find_element_by_css_selector("input[name='username']")
        password_input = self.browser.find_element_by_css_selector("input[name='password']")
        username_input.send_keys(username)
        password_input.send_keys(password)
        login_button = browser.find_element_by_xpath("//button[@type='submit']")
        login_button.click()
        sleep(5)

class HomePage:
    def __init__(self, browser):
        self.browser = browser
        self.browser.get('https://www.instagram.com/')

    def go_to_login_page(self):
        self.browser.find_element_by_xpath("//a[text()='Log in']").click()
        sleep(2)
        return LoginPage(self.browser)

The code is the same except that the home page and the login page are represented as classes. The classes encapsulate the mechanics required to find and manipulate the data in the UI. That is, there are methods and accessors that allow the software to do anything a human can.

One other thing to note is that when you navigate to another page using a page object, it returns a page object for the new page. Note the returned value of go_to_log_in_page(). If you had another class called FeedPage, then login() of the LoginPage class would return an instance of that: return FeedPage().

Here’s how you can put the Page Object Pattern to use:

from selenium import webdriver

browser = webdriver.Firefox()
browser.implicitly_wait(5)

home_page = HomePage(browser)
login_page = home_page.go_to_login_page()
login_page.login("<your username>", "<your password>")

browser.close()

It looks much better, and the test above can now be rewritten to look like this:

def test_login_page(browser):
    home_page = HomePage(browser)
    login_page = home_page.go_to_login_page()
    login_page.login("<your username>", "<your password>")

    errors = browser.find_elements_by_css_selector('#error_message')
    assert len(errors) == 0

With these changes, you won’t have to touch your tests if something changes in the UI.

For more information on the Page Object Pattern, refer to the official documentation and to Martin Fowler’s article.

Now that you’re familiar with both Selenium and the Page Object Pattern, you’ll feel right at home with InstaPy. You’ll build a basic bot with it next.

Note: Both Selenium and the Page Object Pattern are widely used for other websites, not just for Instagram.

How to Build an Instagram Bot With InstaPy

In this section, you’ll use InstaPy to build an Instagram bot that will automatically like, follow, and comment on different posts. First, you’ll need to install InstaPy:

$ python3 -m pip install instapy

This will install instapy in your system.

Note: The best practice is to use virtual environments for every project so that the dependencies are isolated.

Essential Features

Now you can rewrite the code above with InstaPy so that you can compare the two options. First, create another Python file and put the following code in it:

from instapy import InstaPy

InstaPy(username="<your_username>", password="<your_password>").login()

Replace the username and password with yours, run the script, and voilà! With just one line of code, you achieved the same result.

Even though your results are the same, you can see that the behavior isn’t exactly the same. In addition to simply logging in to your profile, InstaPy does some other things, such as checking your internet connection and the status of the Instagram servers. This can be observed directly on the browser or in the logs:

INFO [2019-12-17 22:03:19] [username]  -- Connection Checklist [1/3] (Internet Connection Status)
INFO [2019-12-17 22:03:20] [username]  - Internet Connection Status: ok
INFO [2019-12-17 22:03:20] [username]  - Current IP is "17.283.46.379" and it's from "Germany/DE"
INFO [2019-12-17 22:03:20] [username]  -- Connection Checklist [2/3] (Instagram Server Status)
INFO [2019-12-17 22:03:26] [username]  - Instagram WebSite Status: Currently Up

Pretty good for one line of code, isn’t it? Now it’s time to make the script do more interesting things than just logging in.

For the purpose of this example, assume that your profile is all about cars, and that your bot is intended to interact with the profiles of people who are also interested in cars.

First, you can like some posts that are tagged #bmw or #mercedes using like_by_tags():

 1 from instapy import InstaPy
 2 
 3 session = InstaPy(username="<your_username>", password="<your_password>")
 4 session.login()
 5 session.like_by_tags(["bmw", "mercedes"], amount=5)

Here, you gave the method a list of tags to like and the number of posts to like for each given tag. In this case, you instructed it to like ten posts, five for each of the two tags. But take a look at what happens after you run the script:

INFO [2019-12-17 22:15:58] [username]  Tag [1/2]
INFO [2019-12-17 22:15:58] [username]  --> b'bmw'
INFO [2019-12-17 22:16:07] [username]  desired amount: 14  |  top posts [disabled]: 9  |  possible posts: 43726739
INFO [2019-12-17 22:16:13] [username]  Like# [1/14]
INFO [2019-12-17 22:16:13] [username]  https://www.instagram.com/p/B6MCcGcC3tU/
INFO [2019-12-17 22:16:15] [username]  Image from: b'mattyproduction'
INFO [2019-12-17 22:16:15] [username]  Link: b'https://www.instagram.com/p/B6MCcGcC3tU/'
INFO [2019-12-17 22:16:15] [username]  Description: b'Mal etwas anderes \xf0\x9f\x91\x80\xe2\x98\xba\xef\xb8\x8f Bald ist das komplette Video auf YouTube zu finden (n\xc3\xa4here Infos werden folgen). Vielen Dank an @patrick_jwki @thehuthlife  und @christic_  f\xc3\xbcr das bereitstellen der Autos \xf0\x9f\x94\xa5\xf0\x9f\x98\x8d#carporn#cars#tuning#bagged#bmw#m2#m2competition#focusrs#ford#mk3#e92#m3#panasonic#cinematic#gh5s#dji#roninm#adobe#videography#music#bimmer#fordperformance#night#shooting#'
INFO [2019-12-17 22:16:15] [username]  Location: b'K\xc3\xb6ln, Germany'
INFO [2019-12-17 22:16:51] [username]  --> Image Liked!
INFO [2019-12-17 22:16:56] [username]  --> Not commented
INFO [2019-12-17 22:16:57] [username]  --> Not following
INFO [2019-12-17 22:16:58] [username]  Like# [2/14]
INFO [2019-12-17 22:16:58] [username]  https://www.instagram.com/p/B6MDK1wJ-Kb/
INFO [2019-12-17 22:17:01] [username]  Image from: b'davs0'
INFO [2019-12-17 22:17:01] [username]  Link: b'https://www.instagram.com/p/B6MDK1wJ-Kb/'
INFO [2019-12-17 22:17:01] [username]  Description: b'Someone said cloud? \xf0\x9f\xa4\x94\xf0\x9f\xa4\xad\xf0\x9f\x98\x88 \xe2\x80\xa2\n\xe2\x80\xa2\n\xe2\x80\xa2\n\xe2\x80\xa2\n#bmw #bmwrepost #bmwm4 #bmwm4gts #f82 #bmwmrepost #bmwmsport #bmwmperformance #bmwmpower #bmwm4cs #austinyellow #davs0 #mpower_official #bmw_world_ua #bimmerworld #bmwfans #bmwfamily #bimmers #bmwpost #ultimatedrivingmachine #bmwgang #m3f80 #m5f90 #m4f82 #bmwmafia #bmwcrew #bmwlifestyle'
INFO [2019-12-17 22:17:34] [username]  --> Image Liked!
INFO [2019-12-17 22:17:37] [username]  --> Not commented
INFO [2019-12-17 22:17:38] [username]  --> Not following

By default, InstaPy will like the first nine top posts in addition to your amount value. In this case, that brings the total number of likes per tag to fourteen (nine top posts plus the five you specified in amount).

Also note that InstaPy logs every action it takes. As you can see above, it mentions which post it liked as well as its link, description, location, and whether the bot commented on the post or followed the author.

You may have noticed that there are delays after almost every action. That’s by design. It prevents your profile from getting banned on Instagram.

Now, you probably don’t want your bot liking inappropriate posts. To prevent that from happening, you can use set_dont_like():

from instapy import InstaPy

session = InstaPy(username="<your_username>", password="<your_password>")
session.login()
session.like_by_tags(["bmw", "mercedes"], amount=5)
session.set_dont_like(["naked", "nsfw"])

With this change, posts that have the words naked or nsfw in their descriptions won’t be liked. You can flag any other words that you want your bot to avoid.

Next, you can tell the bot to not only like the posts but also to follow some of the authors of those posts. You can do that with set_do_follow():

from instapy import InstaPy

session = InstaPy(username="<your_username>", password="<your_password>")
session.login()
session.like_by_tags(["bmw", "mercedes"], amount=5)
session.set_dont_like(["naked", "nsfw"])
session.set_do_follow(True, percentage=50)

If you run the script now, then the bot will follow fifty percent of the users whose posts it liked. As usual, every action will be logged.

You can also leave some comments on the posts. There are two things that you need to do. First, enable commenting with set_do_comment():

from instapy import InstaPy

session = InstaPy(username="<your_username>", password="<your_password>")
session.login()
session.like_by_tags(["bmw", "mercedes"], amount=5)
session.set_dont_like(["naked", "nsfw"])
session.set_do_follow(True, percentage=50)
session.set_do_comment(True, percentage=50)

Next, tell the bot what comments to leave with set_comments():

from instapy import InstaPy

session = InstaPy(username="<your_username>", password="<your_password>")
session.login()
session.like_by_tags(["bmw", "mercedes"], amount=5)
session.set_dont_like(["naked", "nsfw"])
session.set_do_follow(True, percentage=50)
session.set_do_comment(True, percentage=50)
session.set_comments(["Nice!", "Sweet!", "Beautiful :heart_eyes:"])

Run the script and the bot will leave one of those three comments on half the posts that it interacts with.

Now that you’re done with the basic settings, it’s a good idea to end the session with end():

from instapy import InstaPy

session = InstaPy(username="<your_username>", password="<your_password>")
session.login()
session.like_by_tags(["bmw", "mercedes"], amount=5)
session.set_dont_like(["naked", "nsfw"])
session.set_do_follow(True, percentage=50)
session.set_do_comment(True, percentage=50)
session.set_comments(["Nice!", "Sweet!", "Beautiful :heart_eyes:"])
session.end()

This will close the browser, save the logs, and prepare a report that you can see in the console output.

Additional Features in InstaPy

InstaPy is a sizable project that has a lot of thoroughly documented features. The good news is that if you’re feeling comfortable with the features you used above, then the rest should feel pretty similar. This section will outline some of the more useful features of InstaPy.

Quota Supervisor

You can’t scrape Instagram all day, every day. The service will quickly notice that you’re running a bot and will ban some of its actions. That’s why it’s a good idea to set quotas on some of your bot’s actions. Take the following for example:

session.set_quota_supervisor(enabled=True, peak_comments_daily=240, peak_comments_hourly=21)

The bot will keep commenting until it reaches its hourly and daily limits. It will resume commenting after the quota period has passed.

Headless Browser

This feature allows you to run your bot without the GUI of the browser. This is super useful if you want to deploy your bot to a server where you may not have or need the graphical interface. It’s also less CPU intensive, so it improves performance. You can use it like so:

session = InstaPy(username='test', password='test', headless_browser=True)

Note that you set this flag when you initialize the InstaPy object.

Using AI to Analyze Posts

Earlier you saw how to ignore posts that contain inappropriate words in their descriptions. What if the description is good but the image itself is inappropriate? You can integrate your InstaPy bot with ClarifAI, which offers image and video recognition services:

session.set_use_clarifai(enabled=True, api_key='<your_api_key>')
session.clarifai_check_img_for(['nsfw'])

Now your bot won’t like or comment on any image that ClarifAI considers NSFW. You get 5,000 free API-calls per month.

Relationship Bounds

It’s often a waste of time to interact with posts by people who have a lot of followers. In such cases, it’s a good idea to set some relationship bounds so that your bot doesn’t waste your precious computing resources:

session.set_relationship_bounds(enabled=True, max_followers=8500)

With this, your bot won’t interact with posts by users who have more than 8,500 followers.

For many more features and configurations in InstaPy, check out the documentation.

Conclusion

InstaPy allows you to automate your Instagram activities with minimal fuss and effort. It’s a very flexible tool with a lot of useful features.

In this tutorial, you learned:

  • How Instagram bots work
  • How to automate a browser with Selenium
  • How to use the Page Object Pattern to make your code more maintainable and testable
  • How to use InstaPy to build a basic Instagram bot

Read the InstaPy documentation and experiment with your bot a little bit. Soon you’ll start getting new followers and likes with a minimal amount of effort. I gained a few new followers myself while writing this tutorial.

If there’s anything you’d like to ask or share, then please reach out in the comments below.


[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]