Package Manager

Installing Packages — Python Packaging User Guide
Installing Python Modules — Python documentation
PyPI - the Python Package Index : Python Package Index
What Is Pip? A Guide for New Pythonistas – Real Python
Python Application Dependency Management in 2018 · Homepage of Hynek Schlawack
Tool recommendations — Python Packaging User Guide

Python look up packages in this order:

user path
site path
system path

On Windows Python loads according to %PATH% variable.

python -c "import site; print(site.getsitepackages())"

Python Module of the Week - Python Module of the Week
Python 3 Module of the Week — PyMOTW 3
dhellmann / PyMOTW-3 — Bitbucket

pipx · PyPI install global dependencies to virtualenv
pipx

A Better Pip Workflow™ — Kenneth Reitz
Pin Your Packages » nvie.com
The Nine Circles of Python Dependency Hell – Knerd – Medium

pip issues:

non-deterministic build (without pinning)
manual update of sub-dependencies (with pinning)
global dependencies (solved with virtual env)
no dependency resolution

How to use GitHub as a PyPi server

my python project setup (+ all tools) (intermediate) anthony explains #396 - YouTube

Anaconda

Anaconda Distribution | Continuum Analytics: Documentation
Presentations & Blog Posts — Conda documentation

introduces the whole root file system
still depends on system libraries
the conda environment could potentially leak
could have used Docker nowadays

Installing pip

pip — documentation
Installation — pip documentation

latest

sudo -v
curl -OL https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py
sudo python ez_setup.py
curl -OL https://bootstrap.pypa.io/get-pip.py
sudo python get-pip.py

`pip` Usage

pip install package
# show all versions on PyPI
pip install package==
# install a particular version
pip install package==0.1.2

# upgrade package
pip install -U package
pip install --upgrade package

# generate requirements.txt
pip freeze > requirements.txt
# install from requirements.txt
pip install --no-cache-dir -r requirements.txt

# Useful for containers:
--disable-pip-version-check
--no-cache-dir

conditions in `requirements.txt`

pip - Is there a way to have a conditional requirements.txt file for my Python application based on platform? - Stack Overflow
PEP 508 – Dependency specification for Python Software Packages | peps.python.org

package1==0.0.1; platform_system != "Windows"
package2==0.0.1; python_version < '3.7' and

Poetry

Poetry - Python dependency management and packaging made easy. uses pyproject.toml, also build and publish packages
python-poetry/poetry: Python dependency management and packaging made easy.

Dependency Management With Python Poetry – Real Python
How I break up with pip and fall in love with poetry my new girlfriend. - DEV Community
How to Create and Use Virtual Environments in Python With Poetry - YouTube

why I will never use python-poetry - YouTube

pipx install poetry


poetry env --bin <path_to_python>
poetry config virtialenvs.in-project true
poetry shell
poetry add <package>

PDM

pdm · PyPI

supports PEP-582, like local node_modules/
uses pyproject.toml, also build and publish packages

PEP 582 – Python local packages directory | peps.python.org

hatch

pypa/hatch: Modern, extensible Python project management

micropipenv

thoth-station/micropipenv: A lightweight wrapper for pip to support requirements.txt, Pipenv and Poetry lock files or converting them to pip-tools compatible output. Designed for containerized Python applications but not limited to them.

do not include resolver
install from lock file
do not create venv
good for container deployment

`pip-tools`

principals adopted by pipenv

pip-tools documentation
jazzband/pip-tools: A set of tools to keep your pinned Python dependencies fresh.

Better Package Management » nvie.com

`pipreqs`

Stop using “pip freeze” for your Python projects | by Prakhar Rathi | Jun, 2022 | Towards Data Science
pipreqs · PyPI

Virtual Environments

use poetry, pyenv virtualenv

Python Virtual Environments: A Primer – Real Python
An Effective Python Environment: Making Yourself at Home – Real Python
how do virtualenvs actually work (advanced) anthony explains #522 - YouTube simple for *nix, explains intricacies in Windows implementation

28.3. venv — Creation of virtual environments — Python documentation
Virtualenv — virtualenv documentation
Deepwalker/pundler: Python bundler-alike alternative to virtualenv

Create Virtual Environment using “virtualenv” and add it to Jupyter Notebook | by B. Chen | Towards Data Science
Python virtualenv and venv do’s and don’ts | InfoWorld
Virtualenv and venv: Python virtual environments explained | InfoWorld
pyvenv vs virtualenv : learnpython

tox --devenv (beginner - intermediate) anthony explains #073 - YouTube

Yelp/aactivator: Automatically source and unsource a project's environment
what is PROMPT_COMMAND? (+aactivator) (intermediate) anthony explains #374 - YouTube

nakulj/auto-venv: Automatically activate virtual environments in fish

TODO: detect and use .activate.sh like aactivator

An Effective Python Environment: Making Yourself at Home – Real Python
Python Virtual Environments – A Primer – Real Python
Using virtual environments with Python ~ The Python Corner

A Minimalist Approach to Python Virtual Environments
willcasey/venvtool

pipenv

I have tried it and have issues with it

pipenv is the recommend package management tool by PyPA and the reference implementation for Pipfile (requirements.txt replacement). It creates virtualenv in ~/.local/share/virtualenvs instead of project folder.

Pipenv: Python Dev Workflow for Humans — pipenv documentation
pypa/pipenv: Python Development Workflow for Humans.
Kenneth Reitz - Pipenv: The Future of Python Dependency Management - PyCon 2018 - YouTube

Pipenv: A Guide to the New Python Packaging Tool – Real Python
Python Versions Management With pyenv
The ABCs of Pipenv and Python Package Management | Dennis O'Keeffe Blog
Managing Application Dependencies — Python Packaging User Guide

Why you should use pyenv + Pipenv for your Python projects !important
The Python virtual environment with Pyenv & Pipenv - DEV Community 👩‍💻👨‍💻
Python Environment 101. How are pyenv and pipenv different and… | by Shinichi Okada | Towards Data Science
Pyenv support broken since version 2018.10.09 ? · Issue #3136 · pypa/pipenv

Issues:

for application only, not for libraries
py2 and py3 cannot co-exist
locked to a single minor version of Python
often reporting version mismatch when running script in other environment
no QA before release
Why is pipenv the recommended packaging tool by the community and PyPA? : Python
Pyenv support broken · Issue #3551 · pypa/pipenv
pip install -e git+https://github.com/pypa/pipenv.git@master#egg=pipenv
pip install --upgrade https://github.com/pypa/pipenv/archive/master.zip
different package versions for different python versions · Issue #2171 · pypa/pipenv
Creating a Pipfile which has different installation instructions depending on operating systems (PyTorch v0.4.1 as an example) - DEV Community

# enter the virtualenv (automatically create Pipfile)
pipenv shell

# force use your host's Python if the Python version mismatch with Pipfile
# warning: the script may use features not available on your host's Python
pipenv --python $(which python3) shell
pipenv --python $(which python2) shell

# install dependencies with `pipenv` instead of pip
# also applicable within pipenv's shell
pipenv install request numpy
# run your script
python script.py

# add dev dependencies
pipenv install pytest --dev
# *also* install dev dependencies
pipenv install --dev

# behaves as pip, for deployment
pipenv install --system --deploy

# update dependencies
pipenv update

# generate requirements.txt for release
pipenv lock -r > requirements.txt
pipenv run pip freeze > requirements.txt
# OR
pipenv lock
pipenv install --ignore-pipfile

# show dependency graphs
pipenv graph
pipenv graph --reverse

pyenv

nvm for Python

Managing Multiple Python Versions With pyenv – Real Python
Better Python version and environment management with pyenv
"Python Versions and Dependencies Made Easy" - Sebastian Witowski (PyConline AU 2021) - YouTube
Python Versions Management With pyenv

pyenv/pyenv-installer: This tool is used to install pyenv and friends.

curl -L https://raw.github.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash
# follow the instructions
pyenv init

sudo apt-get install -y make build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev \
libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python-openssl
# this fix problems when compiling packages
PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.9.5

pyenv virtualenv PROJECT
pyenv activate PROJECT
python -m pip install -r requirements.txt
pyenv deactivate

pyenv install -l
pyenv virtualenvs
pyenv versions
pyenv global VERSION
pyenv local VENV|VERSION

Can't figure out how to build a python that uses the .SO file · Issue #65 · pyenv/pyenv
wxPython Phoenix build fail ("Could not build python extensions"?) in pyenv 3.5.1 on Linux - wxPython Dev - Discuss wxPython

p

easier to use than pyenv, bash and zsh only

qw3rtman/p: Python Version Management Made Simple

virtualenv

it is a little troublesome to setup

Virtualenv create python environment in local folder
virtualenvwrapper documentation create python environment in a centralized folder
Virtualenv vs Virtualenvwrapper · Saurabh Kumar !important

Code4ReferenceTutorial Python virtual environment .
Bob's Blog - Crafting Software: Getting Started with virtualenv and virtualenvwrapper in Python
Use pew, not virtualenvwrapper, for Python virtualenvs
berdario/pew: A tool to manage multiple virtual environments written in pure python

sudo pip install virtualenv
source virtualenv-auto-activate.sh in .bashrc

Usage:

# create env in current directory
virtualenv env
. env/bin/activate
pip install -r requirements.txt  # if any
deactivate

Multiple Python versions

pyenv/pyenv: Simple Python version management
Managing Multiple Python Versions With pyenv – Real Python

How to Install Python 3.7 on Ubuntu 18.04 | Linuxize
How to Install pip for python 3.7 on Ubuntu 18? - Stack Overflow

Packages

http-agents#Python

How to Evaluate the Quality of Python Packages – Real Python

sudo yay -S python-pipenv pyenv pyenv-virtualenv
pip install --user TermRecord ngxtop bpytop
pip install --user ipython black
pip install --user python-pygame

Python Extension Packages for Windows - Christoph Gohlke if you have difficulties building packages
Wheelodex — an Index of Wheels
PyPI Download Stats

JackMcKew/awesome-python-bytes: 😎 🐍 Awesome lists about Python Bytes https://pythonbytes.fm/
Hidden gems: 14 Python libraries too good to overlook | InfoWorld
6 Python libraries every programmer will love | InfoWorld
4 can't-miss Python goodies from Microsoft, Google, Facebook, and Uber | InfoWorld
Top 10 Python Libraries You Should Know | Tryolabs Blog
5 wicked-fast Python frameworks you have to try | InfoWorld
The World of Zope — Zope Project and Community documentation
15 Python Libraries You Should Know About in 2023 - YouTube
Trey Hunner
Splinter documentation
Python 3 Module of the Week — PyMOTW 3

boltons — boltons documentation utilities
mahmoud/boltons: 🔩 Like builtins, but boltons. Constructs/recipes/snippets that would be handy in the standard library. Nothing like Michael Bolton.

zipfile — Work with ZIP archives — Python 3 documentation
Python's zipfile: Manipulate Your ZIP Files Efficiently – Real Python

ActiveState/appdirs: A small Python module for determining appropriate platform-specific dirs, e.g. a "user data dir".

bwasti/cache.py: Python memoization across program runs.

btwael/superstring.py: A fast and memory-optimized string library for heavy-text manipulation in Python

Notify with Python - Towards Data Science

santinic/pampy: Pampy: The Pattern Matching for Python you always dreamed of.

Date times
datetime — Basic date and time types — Python 3 documentation
A Deep Dive Into Date And Time In Python - YouTube
kennethreitz/maya: Datetimes for Humans™
kennethreitz/delegator.py: Subprocesses for Humans 2.0.
Arrow: better dates and times for Python
dateutil/dateutil: Useful extensions to the standard Python datetime features
Pendulum - Python datetimes made easy ❗!important
Python - Pendulum Module - GeeksforGeeks

Web Frameworks

web-dev#WSGI
web-dev#ASGI

A Beginner’s Introduction to Python Web Frameworks
I built the same app 3 times | Which Python Framework is best? Django vs Flask vs FastAPI - YouTube
TypeError/secure.py: Secure 🔒 headers and cookies for Python web frameworks

10 Best Python Web Development Frameworks - ReadWrite
Alternatives, Inspiration and Comparisons - FastAPI

A familiar HTTP Service Framework — responder 1.1.3 documentation

Flask

Welcome | Flask (A Python Microframework)
humiaozuzu/awesome-flask: A curated list of awesome Flask resources and plugins

Flask Apps - Open-Source Web Apps built with automation tools - DEV Community 👩‍💻👨‍💻

Extensions Registry | Flask (A Python Microframework)
Flask-RESTPlus documentation

Python Flask From Scratch - YouTube
Miguel Grinberg - Flask Workshop - PyCon 2015 - YouTube
Miguel Grinberg - Flask at Scale - PyCon 2016 - YouTube
miguelgrinberg/flack: Companion code to my PyCon 2016 "Flask at Scale" tutorial session.
Miguel Grinberg - Microservices with Python and Flask - PyCon 2017 - YouTube
BUILDING MICROSERVICES WITH PYTHON AND FLASK - YouTube
Flask web development with Python - YouTube
Practical Flask Web Development Tutorials - YouTube

Category: Flask - miguelgrinberg.com

Flask project setup: TDD, Docker, Postgres and more - Part 1 - The Digital Cat
Flask project setup: TDD, Docker, Postgres and more - Part 2 - The Digital Cat
Flask project setup: TDD, Docker, Postgres and more - Part 3 - The Digital Cat

Flask Tutorials – Real Python
Welcome to Connexion’s documentation! — Connexion documentation adds OpenAPI
Python REST APIs With Flask, Connexion, and SQLAlchemy – Real Python
Python REST APIs With Flask, Connexion, and SQLAlchemy – Part 2 – Real Python
Python REST APIs With Flask, Connexion, and SQLAlchemy – Part 3 – Real Python
Python REST APIs With Flask, Connexion, and SQLAlchemy – Part 4 – Real Python

Creating REST Services With Flask - DZone Integration
Get started writing your own web services using Python Flask | Opensource.com
peterrus/flask-docker-debugging-vscode-example: Dockerized Flask Development Workflow in VSCode Example

flask-RESTful flask-restless
flask-security flask-sqlalchemy

Category: Flask - miguelgrinberg.com
How Secure Is The Flask User Session? - miguelgrinberg.com
"Flask At Scale" tutorial at PyCon 2016 in Portland - miguelgrinberg.com
The Flask Mega-Tutorial, Part I: Hello, World! - miguelgrinberg.com
Running Your Flask Application Over HTTPS - miguelgrinberg.com
Migrating from Flask-Script to the New Flask CLI - miguelgrinberg.com

Build a CRUD Web App With Python and Flask – Part Two | Scotch
Build a CRUD Web App With Python and Flask – Part Two | Scotch
Build a CRUD Web App With Python and Flask – Part Three | Scotch

Masonite

Introduction - Masonite Documentation
MasoniteFramework/masonite: The Modern And Developer Centric Python Web Framework
Masonite Framework Tutorial Series Part 1 - Installation - DEV Community 👩‍💻👨‍💻
Masonite Framework Tutorial Series Part 2 - Routing - DEV Community 👩‍💻👨‍💻

Masonite Python Framework - New Dashboard Package! - DEV Community 👩‍💻👨‍💻

Async

web-dev#ASGI
fastapi

Sanic Framework one of the first extremely fast Python frameworks based on asyncio, uses uvloop

Falcon | The minimal, fast, and secure web framework for Python

Async I/O

Coroutines and Tasks — Python documentation
asyncio — Asynchronous I/O — Python documentation
asyncio — Asynchronous I/O, event loop, and concurrency tools — PyMOTW 3
pymotw3/source/asyncio at master · reingart/pymotw3
Async function can be start-and-awaited, or started by asyncio.create_taks() which returns a future that can be awaited on
asyncio.to_thread() turns a sync function to an async function

PEP 654 – Exception Groups and except* | peps.python.org
How Exception Groups Will Improve Error Handling in AsyncIO - Łukasz Langa | Power IT Conference - YouTube
Task allows you to seperates the tasks as done and pending queue. You have to make sure the pending tasks are cancelled once an exception is caught.
Then loop and try each task.exception() to gather the results/exceptions.
ExceptionGroup in 3.11 allows tasks to return multiple errors at once. except* statement match subgroup of an ExceptionGroup.
TaskGroup further simplifies the DX.

tasks = [asyncio.create_task(coro(param)) for param in params]
done, pending = await asyncio.gather(*tasks, return_when=asyncio.FIRST_EXCEPTION)

results = await asyncio.gather(
  coro1(), coro2(), coro3(),
  return_exceptions=True
)
exceptions = [ex for ex in results if ex isinstance(ex, Exception)]
# ExceptionGroup new in 3.11
if exceptions:
  raise ExceptionGroup(exceptions)

# TaskGroup new in 3.11
try:
  async with asyncio.TaskGroup as tg:
    for param in params:
      tg.create_task(coro(param))
except* asyncio.TimeoutError:
  ...
except* aiohttp.ClientResponseError:
  ...

timofurrer/awesome-asyncio: A curated list of awesome Python asyncio frameworks, libraries, software and resources

python - How does asyncio actually work? - Stack Overflow
Asynchronous I/O With Python 3
Async IO in Python: A Complete Walkthrough – Real Python
Asyncio : A tutorial for the beginners | KnowPapa
Hands-on Python 3 Concurrency With the asyncio Module – Real Python
Asynchronous Programming in Python | Asyncio (Guide)
How to use asyncio in Python | InfoWorld
3 steps to a Python async overhaul | InfoWorld
Python async/await Tutorial

import asyncio: Learn Python's AsyncIO - YouTube EdgeDB
How To Easily Do Asynchronous Programming With Asyncio In Python - YouTube
Next-Level Concurrent Programming In Python With Asyncio - YouTube

async def find_treasure(start, end):
    global treasure_found
    for i in range(start, end):
        if treasure_found:
            return
        # Await until file is read
  await read_file(i)


async def main():
    tasks = [find_treasure(i, i+count)
            for i in range(0, N, count)]
    await asyncio.gather(
            *tasks
    )

asyncio.run(main())

aio-libs
Tinche/aiofiles: File support for asyncio
theelous3/asks: Async requests-like httplib for python.
vxgmichel/aiostream: Generator-based operators for asynchronous iteration
Welcome to aiochan’s documentation! — aiochan documentation channel

Welcome to AIOHTTP — aiohttp documentation

AnyIO

AnyIO documentation

Rocketry

Why Rocketry? — Rocketry
Rocketry: Insanely Powerful Scheduler | by Mikael Koli | Jul, 2022 | ITNEXT

Unsync

@unsync defines tasks that automatically starts, .result() will wait on the task

Unsync ambient event loop
alex-sherman/unsync: Unsynchronize asyncio

Curio

Curio documentation coroutine-based library for concurrent Python systems programming
A Tutorial Introduction — Curio documentation
dabeaz/curio: Get that harness ready and hold on tight--Curio is gonna take YOU for a walk.

#107: Python concurrency with Curio - YouTube

Trio

Trio: a friendly Python library for async concurrency and I/O
python-trio/trio: Trio – a friendly Python library for async concurrency and I/O

python - What is the core difference between asyncio and trio? - Stack Overflow

Other async libs

Comparing gevent to eventlet | Concurrency in Python
What is gevent? — gevent documentation uses libev

Eventlet Networking Library

MagicStack/uvloop: Ultra fast asyncio event loop. uses uvloop

Socket programming

socket — Low-level networking interface — Python documentation
socketserver — A framework for network servers — Python documentation

Socket Programming in Python (Guide) – Real Python

Distributed Computing

6 Python libraries for parallel processing | InfoWorld
Asynchronous Task Execution In Python

execnet: Distributed Python deployment and communication

Homepage | Celery: Distributed Task Queue

closeio/tasktiger: Python task queue. Because celery is gross.

Welcome to aio-pika’s documentation! — aio-pika documentation wrapper for the aiormq for asyncio and humans

RQ: Simple job queues for Python
rq/rq: Simple job queues for Python
Introducing RQ » nvie.com
Asynchronous Tasks with Flask and Celery | TestDriven.io
Asynchronous Tasks with Flask and Redis Queue | TestDriven.io

Welcome to PyPubSub’s Home Page! — Pypubsub v4.0.3 Documentation
schollii/pypubsub: A Python publish-subcribe library (moved here from SourceForge.net where I had it for many years)

Configs

python-decouple · PyPI load .env
Do You Really Need Environment Variables in Python? | iRead

Dynaconf

Dynaconf
PulpCon 2021 - Dynaconf: What is it, How Pulp uses it, What is coming on v4.0 - YouTube

ConfigParser

configparser — Configuration file parser — Python documentation

import configparser

config = configparser.ConfigParser()
config.read(config_path)
print({section: dict(config[section]) for section in config.sections()})

Database/ORM

sql#ORM (or not)

PEP 249 – Python Database API Specification v2.0 | peps.python.org

SQLModel of the FastAPI family, uses Pydantic

SQLAlchemy - The Database Toolkit for Python
dahlia/awesome-sqlalchemy: A curated list of awesome tools for SQLAlchemy
Data Management With Python, SQLite, and SQLAlchemy – Real Python
How to fix common pitfalls with the Python ORM tool SQLAlchemy | Opensource.com
SQLAlchemy ORM — a more “Pythonic” way of interacting with your database

encode/databases: Async database support for Python. 🗄
collerek/ormar: python async orm with fastapi in mind and pydantic validation

Rasgo
rasgointelligence/RasgoQL: Write python locally, execute SQL in your data warehouse
RasgoQL Brings the Modern Data Stack to Python Users – The New Stack

coleifer/peewee: a small, expressive orm -- supports postgresql, mysql and sqlite
Peewee Tutorial - Tutorialspoint

How to Use PostgreSQL in Python

kennethreitz/records: SQL for Humans™
MagicStack/asyncpg: A fast PostgreSQL Database Client Library for Python/asyncio.

CLI

How to Write User-friendly Command Line Interfaces in Python | by Xiaoxu Gao | Towards Data Science

prompt-toolkit

MasterOdin/crayons: Text UI colors for Python.
tartley/colorama: Simple cross-platform colored terminal text in Python

peterbrittain/asciimatics: A cross platform package to do curses-like operations, plus higher level APIs and widgets to create text UIs and ASCII art animations

amoffat/sh: Python process launching

Prompt

prompt-toolkit/python-prompt-toolkit: Library for building powerful interactive command line applications in Python
python-cmd2/cmd2: cmd2 - quickly build feature-rich and user-friendly interactive command line applications in Python

Textualize/textual: Textual is a TUI (Text User Interface) framework for Python inspired by modern web development.

Introducing Textual uses Rich internally
Building Rich terminal dashboards
Rendering a tree view in the terminal with Python and Rich

TUI

Textualize/rich: Rich is a Python library for rich text and beautiful formatting in the terminal.
Welcome to Rich’s documentation!
Make Your Python CLI Tools Pop With Rich | Hackaday

Overview — Urwid
urwid/urwid: Console user interface library for Python (official repo)

tabulate · PyPI pretty print table and dict

Pytabby: a tabbed menu system for console-based Python programs - DEV Community 👩‍💻👨‍💻

tqdm documentation
tqdm/tqdm: A fast, extensible progress bar for Python and CLI

Parser

Comparing Python Command-Line Parsing Libraries - Argparse, Docopt, and Click - Real Python
Building Beautiful Command Line Interfaces with Python | by Oyetoke Tobi Emmanuel | codeburst

chriskiehl/Gooey: Turn (almost) any Python command line program into a full GUI application with one line

argparse — Parser for command-line options, arguments and sub-commands — Python documentation
Python Argparse Cookbook - mkaz.tech
PEP 389 – argparse - New Command Line Parsing Module | peps.python.org
How to Build Command Line Interfaces in Python With argparse – Real Python
Python Argparse Cookbook – mkaz.blog
Learn Enough Python to be Useful: argparse – Towards Data Science
How to Master Python Command Line Arguments - Towards Data Science
Python argparse regex expression - Stack Overflow type=validator_function

Welcome to Click — Click Documentation
Writing Python Command-Line Tools With Click – dbader.org
Mastering Click: Writing Advanced Python Command-Line Apps – dbader.org
Super Easy Python CLI with Click | CODING w/RICKY
How to Write Python Command-Line Interfaces like a Pro
click-contrib

Typer Click with type hints instead of annotation
tiangolo/typer: Typer, build great CLIs. Easy to code. Based on Python type hints.

Argh: The Natural CLI — argh documentation

mando - CLI interfaces for Humans — mando documentation
rubik/mando: Create Python CLI apps with little to no effort at all!

kbknapp/Clapp-py: A small library for quickly and easily creating command line applications in python.

docopt—language for description of command-line interfaces
Python argparse: Make at least one argument required - Stack Overflow docopt example

Framework

cliff – Command Line Interface Formulation Framework — cliff documentation
Cement Framework
timsavage/pyapp: A Python Application framework - Let us handle the boring stuff!
google/python-fire: Python Fire is a library for automatically generating command line interfaces (CLIs) from absolutely any Python object.

Project Generators

Cookiecutter: Better Project Templates — cookiecutter documentation
cookiecutter/cookiecutter: A command-line utility that creates projects from cookiecutters (project templates), e.g. Python package projects, VueJS projects.

Raphael Pierzina - Kickstarting projects with Cookiecutter - YouTube

Cookiecutter Templates
nvie/cookiecutter-python-cli

bpw1621/ordained: An opinionated template for Python packages.
ORDAINED: The Python Project Template - KDnuggets

GUI

qt#Python
cross-platform-apps-desktop

eBook

Overview — Sphinx documentation
aerkalov/ebooklib: Python E-book library for handling books in EPUB2/EPUB3 and Kindle format -
anqxyr/mkepub: Simple minimalistic library for creating EPUB3 files

SimPy

Overview — SimPy documentation Discrete event simulation for Python
SimPY - YouTube
Meghan Heintz: Launching a new warehouse with SimPy at Rent the Runway | PyData New York City 2019 - YouTube
Basics of Discrete Event Simulation using SimPy in Python

Hydra

Hydra | Hydra
Open-sourcing Hydra for simpler app development - Facebook Engineering

Hydra — A fresh look at configuration for machine learning projects | by PyTorch | PyTorch | Medium

FLOSS Weekly 573 Hydra

Testing

Top 5 Python Frameworks For Test Automation In 2019 | LambdaTest

Getting Started With Testing in Python – Real Python
Continuous Integration With Python: An Introduction – Real Python
wily · PyPI
Testing Your Code — The Hitchhiker's Guide to Python
An Introduction to Unit Testing in Python

Your way out of The Lack of Testing Death Spiral - Pythoscope
mkwiatkowski/pythoscope: unit test generator for Python

Refactoring with tests in Python: a practical example - The Digital Cat

Automated Testing in Python with pytest, tox, and GitHub Actions - YouTube

`pytest`

pytest: helps you write better programs — pytest documentation

funtion tests
load test_<CASE>() functions from arbitary test files *_test.py/test_*.py
Test discovery — pytest documentation

class based

class Test<Name>:
  def test_<CASE>():
    pass

unittest
compatible with unittest test cases
use assert <CONDITION>
test case filtering
@pytest.fixture with context for setup/teardown
plugins

mkdir test/

pytest -v # pytest is too quiet by default
pytest -s # pytest hide your prints, `-s` shows them
pytest -k test_<CASE> # run one test case

pytest --collect-only  # show test cases
pytest --cov
coverage html

anthonywritescode
pytest - anthonywritescode - YouTube
getting started with pytest (beginner - intermediate) anthony explains #518 - YouTube pytest: everything you need to know about fixtures (intermediate) anthony explains #487 - YouTube

Effective Python Testing With Pytest – Real Python

Welcome to pytest-benchmark’s documentation! — pytest-benchmark documentation

How To Write Unit Tests For Existing Python Code // Part 1 of 2 - YouTube
How To Write Unit Tests For Existing Python Code // Part 2 of 2 - YouTube

TDD in Python with pytest - YouTube
TDD in Python with pytest - Part 1 - The Digital Cat
TDD in Python with pytest - Part 2 - The Digital Cat
TDD in Python with pytest - Part 3 - The Digital Cat
TDD in Python with pytest - Part 4 - The Digital Cat
TDD in Python with pytest - Part 5 - The Digital Cat

tox

Welcome to the tox automation project — tox documentation

test with different versions of Python and runtimes
support pytest

introduction to tox (beginner - intermediate) anthony explains #043 - YouTube library
Automating Build, Test and Release Workflows with tox - YouTube

nox

Welcome to Nox — Nox documentation

like tox but use Python script as config

nose

Welcome to nose2 — nose2 documentation

compatible with unittest test cases
parameterization
plugins

`unittest`

unittest — Unit testing framework — Python documentation

Python stdlib
inherit unittest.TestCase and write testing methods

Improve Your Tests With the Python Mock Object Library – Real Python unittest.mock
Python Tutorial: Unit Testing Your Code with the unittest Module - YouTube

Logging

16.6. logging — Logging facility for Python — Python documentation
Logging HOWTO — Python documentation
Logging Cookbook — Python documentation
A guide to logging in Python | Opensource.com
Python Logging: In-Depth Tutorial | Toptal
Logging in Python ~ The Python Corner
Logging — The Hitchhiker's Guide to Python

Logging in Python – Real Python
Python Logging: A Stroll Through the Source Code – Real Python
How to Collect, Customize, and Centralize Python Logs | Datadog

Python logging and rotating files - Stack Overflow
logging - How to create rolling logger in Python - Stack Overflow

python - logger configuration to log to file and print to stdout - Stack Overflow
acschaefer/duallog: Python package to enable simultaneous logging to console and logfile.

def logger_setup(
    filename=None, when="d", interval=1, backup_count=21, level=logging.INFO
):
    formatter = logging.Formatter(
        "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
    )
    # formatter.converter = time.gmtime  # if you want UTC time
    root_logger = logging.getLogger()
    root_logger.setLevel(level)

    if filename is not None:
        file_handler = logging.handlers.TimedRotatingFileHandler(
            filename=filename,
            when=when,
            interval=interval,
            backupCount=backup_count,
        )
        file_handler.setFormatter(formatter)
        root_logger.addHandler(file_handler)

    console_handler = logging.StreamHandler(sys.stdout)
    console_handler.setFormatter(formatter)
    root_logger.addHandler(console_handler)

Python's Built-In 'logging' Module - YouTube

kopensource/colored_logs
apparebit/konsole: Readable, pleasing console output for Python wraps logging API

Do not log

Icecream

gruns/icecream: 🍦 Never use print() to debug again. ❗!important, better print()
Debugging 101: Replace print() with icecream ic() - YouTube

loguru

Delgan/loguru: Python logging made (stupidly) simple ❗!important, better logging
Table of contents — loguru documentation

Python 中更优雅的日志记录方案 loguru | 静觅

Modules

Module corresponds to files, package corresponds to folder.

6. Modules — Python documentation
Python Modules: Creating, Importing, and Sharing
Python Circular Imports
site — Site-specific configuration hook — Python documentation

Python Modules and Packages – An Introduction – Real Python
Python import: Advanced Techniques and Tips – Real Python
The Module Search Path – Real Python
Python behind the scenes #11: how the Python import system works

Module search path:

current directory
environment variable PYTHONPATH
installation dependent list of directories

The resulting search path is accessible in the Python variable sys.path which you can manipulate

PEP 420 – Implicit Namespace Packages | peps.python.org
__init__.py be required, not anymore since 3.3
import * will load __all__: list[str] from __init__.py (package) or module

Python Zip Imports: Distribute Modules and Packages Quickly – Real Python
Live-reloading of Python Modules in the Python REPL / IPython / Jupyter Console - DEV Community 👩‍💻👨‍💻

I don't need __init__.py? PEP 420 and namespace packages (intermediate) anthony explains #420 - YouTube namespace

`importlib`

Writing a Domain Specific Language (DSL) in Python – dbader.org

Packaging/Publishing

TODO: clean up articles using setuptools
TODO: clean up articles using pyproject.toml

python-the-packaging-gradient

#pip-tools
#Poetry
#PDM

A tutorial on packaging up your Python code for PyPI 2017
Python Packaging User Guide — Python Packaging User Guide
Packaging Python Projects — Python Packaging User Guide
Tool recommendations — Python Packaging User Guide
Publishing your own Python package - Towards Data Science 2019, !important
Publishing Python Packages 2022

How to Publish an Open-Source Python Package to PyPI – Real Python
What Are Python Wheels and Why Should You Care? – Real Python
Episode #83: Ready to Publish Your Python Packages? – The Real Python Podcast

How to Build a Complete Python Package Step-by-Step - YouTube
"Publishing well-formed Python packages" - Julin S (PyConline AU 2020) - YouTube

packaging - anthonywritescode - YouTube

What to do when you botch a release on PyPI

Welcome to twine’s documentation! — twine documentation THE recommended publishing tool for PyPI
pypa/twine: Utilities for interacting with PyPI

The difference between setup.py (pyproject.toml) and requirements.txt (Pipfile) · Issue #27 · pypa/pipfile
setup.py vs requirements.txt · caremad
Pipfile vs setup.py — pipenv documentation

The Many Layers of Packaging — Sedimental

Python Wheels
pypa/wheel: The official binary distribution format for Python
pypa/manylinux: Python wheels that work on any linux (almost)
pypa/python-manylinux-demo: Demo project for building Python wheels for Linux with Travis-CI

bump2version · PyPI Version-bump your software with a single command!

`setuptools`

Paul Ganssle - Why you shouldn't invoke setup.py directly history
use modern pyproject.toml based build tools

distutil is DEPRECATED by setuptools

Welcome to Setuptools’ documentation! — setuptools documentation
Python Entry Points Explained
The Sheer Joy of Packaging! — The Joy of Packaging documentation 2018
Packaging and distributing projects — Python Packaging User Guide uses twine, setup.py and setup.cfg
Building a Python API for Raspberry Pi hardware - Speaker Deck 2015

python setup.py sdist # source distribution, allows package manager to build Wheel, no control
python setup.py bdist_wheel --universial # Wheels distribution (binary), for pure Python code
python setup.py install

Mahmoud Hashemi, "The Packaging Gradient", PyBay2017 - YouTube slide
An Introduction to Python Packages for Absolute Beginners - By Ramit Mittal

Pipenv: A Guide to the New Python Packaging Tool – Real Python Using setuptools with pipenv
Put dependencies in setup.py instead of Pipfile, use pipenv install -e to install package locally.

pypa/sampleproject: A sample project that exists for PyPUG's "Tutorial on Packaging and Distributing Projects"
requests/setup.py at master · requests/requests setuptools
numpy/setup.py at master · numpy/numpy distutils?

`pyproject.toml`

Defines pyproject.toml, decouples from setuptools to allow multiple build tools.

PEP 517 – A build-system independent format for source trees | peps.python.org entry points for the build
PEP 518 – Specifying Minimum Build System Requirements for Python Projects | peps.python.org build dependencies
PEP 631 – Dependency specification in pyproject.toml based on PEP 508 | peps.python.org

What the heck is pyproject.toml?
Clarifying PEP 518 (a.k.a. pyproject.toml)

Build tools

build backend: knows how to take a source tree and turn it into a distributable artifact — a source distribution or a wheel (e.g.: setuptools or flit)
build frontend: orchestrates the build environment and invokes the build backend, possibly creates environment for the build (e.g.: pip or build, test runners like tox or nox)

build PEP 517 frontend

Ruff all-in-one tool, used by many
Goodbye to Flake8 and PyLint: faster linting with Ruff

takluyver/flit: Simplified packaging of Python modules backend, for pure Python module
Flit — Flit documentation

dholth/enscons build extensions
enscons · PyPI

Welcome to the tox automation project — tox documentation

cibuildwheel · PyPI
Home - cibuildwheel

David-OConnor/pyflow: An installation and dependency system for Python

ofek/hatch: A modern project, package, and virtual env manager for Python build, test and upload with one CLI
About - Hatch

PyBuilder — Usage Documentation pbr, not using pyproject.toml
OpenStack Docs: pbr Usage
Build wheel with a minimal setup.py and setup.cfg.
Building a Python package, and a docker image via Pipenv

Packaging Application/Freezing

The act of bundling all dependencies, and optionally to make a standalone binary, is called freezing in Python

Packaging Archives - The Mouse Vs. The Python
Episode #245 Python packaging landscape in 2020 - [Talk Python To Me Podcast]
tryexceptpass - 4 Attempts at Packaging Python as an Executable
Convert Python to exe - copyassignment.com

schmir/bbfreeze: UNMAINTAINED
A bbfreeze Tutorial – Build a Binary Series! | The Mouse Vs. The Python

py2exe and py2app are Windows and OSX specific solution

osnap
pynist
nuitka

Briefcase
Nuitka the Python Compiler — Nuitka the Python Compiler documentation

linkedin/shiv: shiv is a command line utility for building fully self contained Python zipapps as outlined in PEP 441, but with all their dependencies included.

pantsbuild/pex: A library and tool for generating .pex (Python EXecutable) files

shebang + zip file
requires system Python
OS specific (can be built for cross-platform)
similar to wheels

PyInstaller

Welcome to PyInstaller official website
PyInstaller Manual — PyInstaller documentation
pyinstaller/pyinstaller: Freeze (package) Python programs into stand-alone executables

brentvollebregt/auto-py-to-exe: Converts .py to .exe using a simple graphical interface
How to Create Executable Applications in Python | Tom's Hardware

How to deploy PyQt, Keras, Tensorflow apps with PyInstaller - YouTube
PyInstaller did what cx_Freeze failed, with 300MB less size.
This is done possibly with tree shaking.

Introducing PyInstaller | Linux Journal
Using PyInstaller to Easily Distribute Python Applications – Real Python

FAQ · pyinstaller/pyinstaller Wiki
Recipes · pyinstaller/pyinstaller Wiki

When Things Go Wrong — PyInstaller documentation

When debugging:

check build\{NAME}\ for logs

check debug messages:

pyinstaller --clean -y --log-level=DEBUG specfile 2> build.log

run bundled app at console to see error log
use onedir to view bundle structure easily, also pyi-archive_viewer

setuptools 45 breaks PyInstaller ("Failed to execute script pyi_rth_pkgres"), downgrade to 44 until this is fixed
setuptools 45.0.0 may cause PyInstaller 3.3 packaged executable fail to launch · Issue #1963 · pypa/setuptools

use latest version
pip install https://github.com/pyinstaller/pyinstaller/archive/master.zip

pip install --upgrade 'setuptools<45.0.0'

Hooks

Understanding PyInstaller Hooks — PyInstaller documentation
pyinstaller/PyInstaller/hooks at master · pyinstaller/pyinstaller

Trouble with Tensorflow 2.0 · Issue #4400 · pyinstaller/pyinstaller hook-tensorflow_core.py
Python, Pandas, & PyInstaller — Watch Out! - Nathan Benton - Medium
Pyinstaller hooks — Kivy 1.11.1 documentation

cx_Freeze

cx_Freeze
Welcome to cx_Freeze’s documentation!
cx_Freeze

Your source code is put in lib/library.zip, some tricks is needed to add data to the bundle.
This is the bundled folder structure:

build/exe.linux-x86_64-3.7/
- lib/library.zip
- ui/
- main

The aim is to load asset from ui/ from scripts packaged in lib/library.zip.

# freeze helper
from os import path
import sys

if getattr(sys, "frozen", False):
    # The application is frozen
    print(f"frozen app, exec='{sys.executable}'")
    ASSET_DIR = path.dirname(sys.executable)
else:
    # The application is not frozen
    # Change this bit to match where you store your data files:
    ASSET_DIR = path.dirname(__file__)

# setup.py
import sys
from cx_Freeze import setup, Executable

# Dependencies are automatically detected, but it might need
# fine tuning.
includeFiles = ["ui"]
buildOptions = dict(packages=[], excludes=[], include_files=includeFiles)

base = "Win32GUI" if sys.platform == "win32" else None

executables = [Executable("main.py", base=base)]
setup(
    name="demo",
    version="1.0",
    description="",
    options=dict(build_exe=buildOptions),
    executables=executables,
)

python setup.py build

Linters

Python Code Quality Authority
An Introduction to the PyCQA
Python Code Quality: Tools & Best Practices – Real Python !important
7 Python libraries for more maintainable code | Opensource.com

Auto formatters for Python 👨‍💻🤖 - 3YOURMIND-Tech - Medium

ambv/black: The uncompromising Python code formatter
The uncompromising code formatter — Black documentation

asottile/reorder_python_imports: Rewrites source to reorder python imports

--py37-plus --add-import "from __future__ import annotations"

asottile/pyupgrade: A tool (and pre-commit hook) to automatically upgrade syntax for newer versions of the language.
Python Type Hints - How to Upgrade Syntax with pyupgrade - Adam Johnson

google/yapf: A formatter for Python files

Flake8: Your Tool For Style Guide Enforcement
Pylint User Manual generates too much warning
pycodestyle’s documentation pycodestyle is the new pep8

Pylint vs flake8 detailed comparison as of 2017 - Slant
python - PyLint, PyChecker or PyFlakes? - Stack Overflow

Refactoring Python Applications for Simplicity – Real Python
Python Code Quality: Tools & Best Practices – Real Python

What is wily? — wily develop documentation
Anthony Shaw - Wily Python: Writing simpler and more maintainable Python - PyCon 2019 - YouTube

Welcome to Radon’s documentation! — Radon documentation
rubik/radon: Various code metrics for Python code

Pysa: Open Source static analysis for Python code - Facebook Engineering

Type hints/Typing

Python 3.5 adds type hints to functions, 3.6 extends that for variable.
3.10 support type guards/type narrowing
Type Guard: boolean value that have implication to type of variables of union type
PEP 647 – User-Defined Type Guards | peps.python.org

PEP 484 – Type Hints | peps.python.org
PEP 561 – Distributing and Packaging Type Information | peps.python.org .pyi, py.typed
Our journey to type checking 4 million lines of Python | Dropbox Tech Blog

Python Type Checking (Guide) – Real Python !important
the state of type hints in Python
Stanford Seminar - Optional Static Typing for Python - YouTube
Types at the edges in Python – MeadSteve's Dev Blog
Modernize Your Sinful Python Code with Beautiful Type Hints | by Eirik Berge | Jul, 2021 | Towards Data Science
Typechecking with a Python Library That Has No Type Hints ·
python 学习笔记：typing 和 pydantic | 小猴子 jerry

typing — Support for type hints — Python documentation
Type hints cheat sheet (Python 3) — Mypy documentation
How to Use Static Type Checking in Python 3.6 – Adam Geitgey – Medium
Using Python's Type Annotations - DEV Community 👩‍💻👨‍💻
Python type annotations | Caktus Group
python/typeshed: Collection of library stubs for Python, with static types

"advanced-python-typing" series
1-minute guide to real constants in Python - DEV Community 👩‍💻👨‍💻

5 Reasons Why You Should Use Type Hints In Python - YouTube
Type-Checking Python Programs With Type Hints and mypy - YouTube
Carl Meyer - Type-checked Python in the real world - PyCon 2018 - YouTube

Mypy can be used to do static type checking on type hints.
mypy - Optional Static Typing for Python
Welcome to Mypy documentation! — Mypy documentation
Introducing Mypy, an Experimental Optional Static Type Checker for Python | Linux Journal
Python's Mypy--Advanced Usage | Linux Journal
Mypy improves static type-checking for big Python apps | InfoWorld

Pyre · A performant type-checker for Python 3 Facebook
Overview · Pyre
Pieter Hooimeijer - Types, Deeper Static Analysis, and you - PyCon 2018 - YouTube

microsoft/pyright: Static type checker for Python Microsoft

google/pytype: A static type analyzer for Python code Google

Instagram/MonkeyType: A system for Python that generates static type annotations by collecting runtime types

advanced-python-typing (7 Part Series)
1-minute guide to real constants in Python - DEV Community 👩‍💻👨‍💻
Simple dependent types in Python - DEV Community 👩‍💻👨‍💻
Python exceptions considered an anti-pattern - DEV Community 👩‍💻👨‍💻

prechelt/typecheck-decorator: For Python3, e.g. @typecheck add_count(count: int, when: any(datetime, timedelta) = datetime.now)