Archive

Streamlit is a python library to build data apps. This is similar to observable, and evidence.

With uv, it is easy to see what streamlit looks like -- uv run --with streamlit streamlit hello -- this launches a demo app with a couple different visualization screens.

This is more along the lines of "UI controls that allow you to change the parameters of a chart/visualization" than evidence's "Dynamic report" approach.

Streamlit is a pythonic middle ground between reporting oriented evidence and a very visualization oriented observable.

another way to look at these three: streamlit is python, observable is javascript, and evidence is sql.

UV is a python package and project manager, written in rust.

See how to Switching from pyenv to uv; via HN.

---

Using uv to run arbitary stuff. .. inspect a data file using pandas and apache-arrow.

$ uv run --with pandas --with pyarrow python
>>> import pandas as pd
>>> df = pdf.read_parque("blah")
>>> df.shape
>>> df.info()
>>> 

The PEP-723 lays out the intentions and how the metadata that can be embedded into python scripts to assist launchers, IDEs etc to interact with the script. uv was the first tool (to my knowledge) that brought this into wide attention.

Since then many python ecosystem tools have started to leverage this.

• Marimo -- Serializing package requirements in marimo notebooks
• nanodjango uses this illustrate how a single file can used to run a web application with the help of PEP-723 inline script metadata.

Scikit-learn provides libraries for - classification, regression, clustering, dimensionality reduction, model selection, preprocessing etc.

Scitkit-learn can be (should be) installed into a python virtualenv with pip install -U scikit-learn (the -U flag stands for "upgrade"). This installation in addition may requireother popular python data science libraries like numpy, pandas, matplotlib, cython, polars, .. etc for full functionality. The basic install only installed the following.

# 2024-08-27
$ pip freeze
joblib==1.4.2
numpy==2.1.0
scikit-learn==1.5.1
scipy==1.14.1
threadpoolctl==3.5.0

See also: pytorch, and tensorflow

Django from first principles

Static site generator for Django - Django Distill

While making https://thesummarist.net, I came across https://django-distill.com, which is THE BEST way to create static sites, bar none. You just make a standard, dynamic Django site, then you run a command and boom! It's static. I love it. -- via Stravos

Django SQLite Production Config

# yourproject/settings.py
DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.sqlite3",
        "OPTIONS": {
            "transaction_mode": "IMMEDIATE",
            "timeout": 5,  # seconds
            "init_command": """
                PRAGMA journal_mode=WAL;
                PRAGMA synchronous=NORMAL;
                PRAGMA mmap_size = 134217728;
                PRAGMA journal_size_limit = 27103364;
                PRAGMA cache_size=2000;
            """,
        },
    }
}

Write complete web applications in Python without writing javascript. There are a large class of problems for which this is an excellent solution, without having to dip into the javascript tarpit.

The advantage of something like this over using templates is that you can think of "widgets" vs "templates".

Some frameworks are:

Reflex (formerly known as pynecone). Uses FastAPI + React.

NiceGUI. Uses FastAPI + Vue.js + Quasar, tailwind.

Pydantic/FastUI. Uses React. still early in development.

Gradio -- "Build & Share Delightful Machine Learning Apps"

(Moritz et al. 2018) PDF

Ray unifies task-parallel and actor programming models in a single dynamic task graph and employs a scalable architecture enabled by the global control store and a bottom-up distributed scheduler. The programming flexibility, high throughput, and low latencies simultaneously achieved by this architecture is particularly important for emerging artificial intelligence work- loads, which produce tasks diverse in their resource requirements, duration, and functionality.

Learning Ray by Max Pumperla, Edward Oakes, and Richard Liaw. ( PDF ).

In short, Ray sets up and manages clusters of computers so that you can run distributed tasks on them. A Ray Cluster consists of nodes that are connected to each other via a network. You program against the so-called driver, the program root, which lives on the head node. The driver can run jobs, a collection of tasks, that are run on the nodes in the cluster. Specifically, the individual tasks of a job are run on worker processes on worker nodes.

This cluster can also be local, on your laptop. The default number of worker processes is the number of CPUs available on your machine.

Github - https://github.com/ray-project/ray

$ pip install "ray[rllib, serve, tune]==2.10.0"

import ray
ray.init()

Libraries:

• data processing -- apache-arrow
• model training -- TensorFlow, pytorch
• hyperparameter tuning
• model serving

News:

• Amazon’s Exabyte-Scale Migration from Apache Spark to Ray on Amazon EC2 | AWS Open Source Blog

References

• Moritz, Nishihara, Wang, Tumanov, Liaw, Liang, Elibol, Yang, Paul, Jordan, Stoica (2018). Ray: A Distributed Framework for Emerging AI Applications.

Cog: A Code Generation Tool Written in Python; this is apparently an old tool (blog date 2004), that I had never heard about before. But, the idea of code generation is a useful one, and something Lisps do really well, and is a core part of their identity "code is data".

nim among other languages also have this "metaprogramming" ability.

Do not misuse Jupyter, understand the limitations -- "Notebooks are McDonalds of Code."

Books

Teaching and Learning with Jupyter An open book about Jupyter and its use in teaching and learning.

Tools

seeM/Jim: Jim is a simple, beautiful Jupyter notebook editor for macOS (last commit in 2023)

nbdev – Create delightful software with Jupyter Notebooks Write, test, document, and distribute software packages and technical articles — all in one place, your notebook.

voila -- Voilà allows you to convert a Jupyter Notebook into an interactive dashboard that allows you to share your work with others.

Alternatives / Improvements

marimo "is a reactive notebook for Python that models notebooks as dataflow graphs: run a cell or interact with a UI element, and marimo automatically runs affected cells, keeping code and outputs consistent and preventing bugs before they happen. Every marimo notebook is stored as pure Python, executable as a script, and deployable as an app." ;Lessons learned reinventing the Python notebook | marimo

nteract "NATIVE INTERACTIVE NOTEBOOKS"

In other languages

See also: srcbookdev/srcbook: JS/TS notebook for typescript.

Clerk in clojure is another alternative to jupyter note books.

Livebook.dev is for elixir.

See Data Apps for frameworks.

Paul Snively described it as: If you think of it as "an outliner front-end for noweb," I think that captures 80% of it.

Leo editor running on @nixos_org on MacOS using @UTMapp

this is what using a program written in "cross platform" GUI toolkit often looks like 🙄 pic.twitter.com/SjHVYh7Gao

— Pradeep Gowda (@btbytes) May 11, 2023

Mojo Homepage: https://www.modular.com/mojo

Jeremy Howard demo for Mojo launch - YouTube

fast.ai - Mojo may be the biggest programming advance in decades

metaobject blog: Mojo is a much better "Objective-C without the C" than Swift ever was; Jun 2023.

Modular: Using Mojo🔥 with Python🐍

2023 LLVM Dev Mtg - Mojo 🔥: A system programming language for heterogenous computing - YouTube; Nov 2023.

Use locally built standard library in Mojo; Mar 2024.

Watchfiles is a file watching and code reload in python, written in rust.

Usage:

# pip install watchfiles
from watchfiles import watch

for changes in watch('./path/to/dir'):
    print(changes)

Also supports async via awatch:

import asyncio
from watchfiles import awatch

async def main():
    async for changes in awatch('/path/to/dir'):
        print(changes)

asyncio.run(main())

Alternatives

• entr

Top level notes

• Pandas Documentation Homepage
• Installing (2.0) -- pip install pandas==2.0
  • installing pandas will also install numpy
• Pandas helps in working with tabular data (databases, spreadsheets)
• Helps = explore, clean and proess
• Anything you can do with SQL you can also do with Pandas.
  • Group by operations
• Summary Statistics. eg., mean, median, std..
• The data table is called a DataFrame
• pandas can read (read_ methods) from various file formats like csv, xls, parquet, hdf5, json, sql (db) and generate output (to_ methods) in the above formats.
• matplotlib is used to generated plots from data tables.
• split-apply-combine approach. Hmm.. map-reduce?
• melt() to convert from wide -> long/tidy form and pivot() to convert from long to wide format.
• combine multiple tables using concat() function for column-wise or row-wise joining
• for database like joining/merging, use the merge function. pd.merge(tbone, tbtwo, how="left", left_on="col1", right_on="col2"). also merge_asof
• pandas supports inner, outer, right and left joins.
• use shape() to find the rowxcolumn shape.
• TODO: how vectorization is done in pandas
• TODO: where numpy is faster than pandas
• TODO: Pyspark with pandas

Quick intro

import numpy as np
import pandas as pd

# create new Series
s = pd.Seres([1,2,3, pd.nan, 8, 121])
# pd.nan = not a number

# pd.date_range("<YYYYMMDD", periods=n) produces a
# a date range starting at YYYYMMDD for the period `n`

# dataframes are created with

df = pd.DataFrame(nparray, index=<indexvec>, columns=list(names))
# the nparray is a 2-d numpy array of mxn dimensions

# the df can also be created by passing a dict with keys as the
# column names and the value as the series.

Articles

• Practical SQL for Data Analysis | Haki Benita in comparison to pandas.

wilsonzlin/minify-html : Extremely fast and smart HTML + JS + CSS minifier, available for Rust, Deno, Java, Node.js, Python, Ruby, and WASM

The Python library on pypi.

Using the Python library:

# code/example_minifyhtml.py
#!/usr/bin/env python

import minify_html

with open("output/index.html") as f:
    t = f.read()
    print(t)
    o = minify_html.minify(
        t,
        do_not_minify_doctype=True,
        keep_spaces_between_attributes=True,
        ensure_spec_compliant_unquoted_attribute_values=True,
    )
    print(o)

JAX is a Python library which augments numpy and Python code with function transformations which make it trivial to perform operations common in machine learning programs. Concretely, this makes it simple to write standard Python/numpy code and immediately be able to

Compute the derivative of a function via a successor to autograd Just-in-time compile a function to run efficiently on an accelerator via XLA Automagically vectorize a function, so that e.g. you can process a “batch” of data in parallel. learn more ... "You don't know Jax" (2019)

Jax's website - google/jax: Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more

JAX and Equinox: What are they and why should I bother?

My Python Stuff

• My PyCon 2010 notes
• See posts and notebooks tagged #python
  • My Happy Python Workflow
  • Getting Out of the Programming Goldilocks Zone

Meta

• Awesome Python

Advanced Python

dabeaz-course/python-mastery: Advanced Python Mastery (course by David Beazley)

Books

• Fundamentals of Python programming (PDF) -- "... introduction to computer programming textbooks are available in PDF form for free download. University and high school students and self learners around the world have found them helpful. See the terms of use in the preface of each book."

• Intermediate Python -- "Fluent python" by Luciano Ramalho.

Learning sources

• Tiny-Python-3.6-Notebook/python.rst at master · mattharrison/Tiny-Python-3.6-Notebook

"This repository contains the text for the Tiny Python 3.6 Notebook. it is a notebook containing curated examples for Python 3 as well as the new features found in Python 3.6."

Internals

• How virtual environments work
• CPython Dynamic Dispatch Internals: How a + b Works?

Developer Experience (DevEx)

Make Python DevEx -- this is very close to how I choose to do Python development. good read. I don't have such an extensive Makefile setup.

Debugging

• Debugging cpython with gdb

• Martin Heinz - "Ultimate Guide to Python Debugging" 2020-06-02. Talks a lot about logging, but there is some interesting bits about using __repr__ and __missing__ dunder methods.

Profiling

• benfred/py-spy: Sampling profiler for Python programs

• Scalene: a high-performance, high-precision CPU and memory profiler for Python has interleaved cross-language profiling.

• Pyflame: A Ptracing Profiler For Python — Pyflame 1.4.0 documentation

Web

• emmett-framework/granian: A Rust HTTP server for Python applications
• Robyn - An async Python Framework written in Rust.

Multiprocessing

• Stabbing yourself with a fork() in a multiprocessing.Pool full of sharks

Async

• Concurrency and async / await - FastAPI

Sublime Text

• Flake8Lint is infuriating but important plugin for sublime text. Flake8 Rules

Alternate Python implementations

• Zippy ; Slides, paper

To Install Zippy:

hg clone https://bitbucket.org/ssllab/zippy
cd zippy
TBD

Libraries

• Deciphering Glyph :: Python Packaging Is Good Now; August 2016.
• Cookiecutter -- A command-line utility that creates projects from cookie cutters (project templates). See Example using cookie cutter for pandoc presentations.
• Python modules you should know
• 20 Python Libraries You Aren’t Using (But Should) [PDF] by Caleb Hattingh on O'Reilly. (broken link; sep 2024)
• Peep. A "pip install" that is cryptographically guaranteed repeatable
• imutils -- A series of convenience functions to make basic image processing operations such as translation, rotation, resizing, skeletonization, and displaying Matplotlib images easier with OpenCV and Python.
• The Python Graph Gallery – Visualizing data – with Python
• Getting the Most out of Sqlite3 with Python; See dataset and apsw.
• mahmoud/awesome-python-applications: 💿 Free software that works great, which also happens to be open-source Python.

Data science libraries

• agate 1.6.0 — agate 1.6.0 documentation
• dataset: databases for lazy people
• Python Itertools: For a faster and memory efficient code – Kanoki

Conferences

• PyData (NYC 2013).
• Videos.

Python3

• The key differences between Python 2.7.x and Python 3.x with examples
• "the not-so-simple intricacies of Python2->Python3" - oilshell comments on Python 3 and Firefox 57 (an observation)

Pypy

• PyPy Status Blog: The First 15 Years of PyPy — a Personal Retrospective

Package management

• Python packages with pyproject.toml and nothing else | Simon Willison’s TILs; Jul 2023. Discovered it via My User Experience Porting Off setup.py

• Python Project Tooling explained — chobeat; Apr 2019. lobste.rs

  • Poetry
  • Pipsi
  • pyenv (polyglot alternate is mise; is a replacement for pyenv, rbenv, asdf, nvm, etc.,)
  • not mentioned above, but see also hatch and pdm

• Use pew, not virtualenvwrapper, for Python virtualenvs

• Use twine, and wheel.

• Follow the packaging tutorial

• Use TestPyPi

• Short summary of commands:

python setup.py sdist
python setup.py bdist_wheel
twine upload --repository-url https://test.pypi.org/legacy/ dist/*

• uv: An Extremely Fast Python Package Manager - YouTube, talk by Charlie Marsh of Astral.sh

Project Managers

Do not use requirements.txt - quanttype -- instead choose Poetry, Hatch or pdm.

Poetry

I have been using poetry for a while.

Poetry uses the pyproject.toml standard, but it does not follow the standard specifying how metadata should be represented in a pyproject.toml file (PEP 621), instead using a custom [tool.poetry] table. This is partly because Poetry came out before PEP 621. -- from pdm's documentation

Hatch

• Standardized build system with reproducible builds by default
• Robust environment management with support for custom scripts
• Easy publishing to PyPI or other indexes
• Version management
• Configurable project generation with sane defaults
• Responsive CLI, ~2-3x faster than equivalent tools

pdm

Unlike Poetry and Hatch, PDM is not limited to a specific build backend; users have the freedom to choose any build backend they prefer.

Links

• Solving the Water Jug Problem from Die Hard 3 with TLA+ and Hypothesis
• Writing beautiful intelligent python -- talk by Raymond Hettinger in Pycon 2015.
• profiling - Python profiler. HN; Aug 2015
• vprof - Visual Profiler; flame charts, memory usage etc.,
• Pyparallel -- "An experimental, proof-of-concept fork of Python 3 designed to optimally exploit multiple CPU cores, fast SSDs, NUMA architectures and 10Gb+ Ethernet networks. Removes the limitation of the Global Interpreter Lock (GIL) without needing to remove it at all."
• Replacing Python: Candidates. A survey of various languages while keeping these considerations in mind: speed, binary compatibility, safety, diagnostics, ease of writing, shared libs, static typing, bounds of privilege, mutability, C interoperability and asynchronous code. OCaml appears to be the winner. This is written by the author of 0install.
• Setting up Emacs for python development including Pymacs, pyflakes, pep8, python-mode.el, python.el etc.,. Pymacs has detailed documentation (31pp) on their website.
• Python Tips -- Intermediate Python
• Nov 2015; Python for High performance scintific computing (disussion)
• Python: Generators, Coroutines, Native Coroutines and async/await | Abu Ashraf Masnun -Parallelizing a Python Function for the Extremely Lazy - tdhopper.com
• The Little Book of Python Anti-Patterns — Python Anti-Patterns documentation

2019

Via My Python Development Environment, 2020 Edition | Jacob Kaplan-Moss:

• pipx · PyPI
• Poetry - Python dependency management and packaging made easy.
• VisiData