here you goThings I’ve made, read and saved — shared here.

Archive

9,692 entries: 92 from X, 8,929 from Pinboard, saved 1998 to 2026.

Browse by tag, site, author, folder or language

Tags

Sites

Authors

Folders

Languages

Tags and folders

Showing 4,001–4,050 of 9,692

2020

Unikernels

Talks

Blogs

Articles

Projects

HermitCore

LightVM

LightVM - Systems and Machine Learning

We achieve lightweight VMs by using unikernels for specialized applications and with Tinyx, a tool that enables creating tailor-made, trimmed-down Linux virtual machines. LightVM can boot a VM in 2.3ms which comparable to fork/exec on Linux (1ms), and two orders of magnitude faster than Docker. On a somewhat slower server with 64 cores, LightVM can pack thousands of VMs with memory and CPU usage comparable to that of processes

Hermitux

HermiTux | A binary-compatible unikernel; Apr 2019.

HermiTux is binary-compatible with Linux: it can run native Linux executables.

Although being a proof-of-concept, HermiTux supports multiple compiled (C, C++, Fortran) and interpreted (Python, LUA) languages. It provides binary analysis and rewriting techniques to optimize system call latency and modularize a kernel in the presence of unmodified binaries. It supports statically and dynamically linked programs, different compilers and optimization levels. HermiTux also provides basic support for multithreading, debugging and profiling.

Unik

solo-io/unik: The Unikernel Compilation and Deployment Platform

UniK (pronounced you-neek) is a tool for compiling application sources into unikernels (lightweight bootable disk images) rather than binaries. UniK runs and manages instances of compiled images across a variety of cloud providers as well as locally on Virtualbox. UniK utilizes a simple docker-like command line interface, making building unikernels as easy as building containers.

Unik supports the following unikernel types:

  • Rumpkernels
  • OSv
  • IncludeOS
  • MirageOS

And supports the following providers:

  • Virtualbox
  • AWS
  • Google Cloud
  • vSphere
  • Xen etc.,

More info on Unik:

Unikraft

Related

Links

... the symmetric multiprocessing story - Unikernels don't have such a story. They are inherently single core. ... I just couldn't ignore was that Linux let you configure it into very small and secure configurations with full SMP support AND you could run ANY existing Linux languages or applications on those tiny systems. Unikernels generally seemed to need special builds and toolchains and configuration systems and recompilations to get them to run applications. AUTHORS's takeaway -- "I had come to see that it was possible to build very very small Linux systems, compatible with all Linux software, with all the security advantages of Unikernels, but with the benefit of symmetric multiprocessing and all the Linux driver support and community support."

Papers

News

Tutorials

dysinger/restack: Full Stack ReasonML Tutorials

This is a tutorial series on using the ReasonML language to build MirageOS unikernels and user-facing applications with tools like Reason-React. Using the same language for the entire stack is appealing. You’ll likely be able to stay in one editor, become very familiar with the syntax, and share common code between the layers of your application.

notes

Github Gists

Gist -- http://gist.github.com/ is a small code hosting service provided by github. It is a convenient place to store:

  • one-off scripts
  • snippets of code that is reusable.
  • Short HOW-TO notes that can save you time the next time around
  • Funny, memorable, poignant texts that you got as a forward etc.,

See my Gist Index and this is a better index (gist.aviperl.me), to see rendered files.

By others -- vsouza/awesome-gists: A collection of amazing gists

gist hostby simon willison is a tool that lets you see a browser-rendered version of an HTML page that you have saved to a Gist.

Sublime text

I use sublimetext's gist package to paste, edit and manage the gists directly from inside the sublime text editor. This is a very nifty thing I found out in Dec 2018.

Databases

Shell

Python

C

Web

Gist related projects

Others gists

notes#github#web

Lua Programming Language

Learn

Lua associated languages

Pallene + Lua. Paper .Found via janet lang's gitter channel. It is a "Friendly Fork" of Titan Language. Pallene is a moon of saturn?

Terra -- low level lua like language.

teal; The compiler for Teal, a typed dialect of Lua

Luau is a fast, small, safe, gradually typed embeddable scripting language derived from Lua.

Links and references

Cool Lua stuff

2020-06-04 oasislinux/oasis: a small statically-linked linux system; uses lua for setup, building, package spec etc.,

Mako Server This tiny, ready-to-run application server packs a punch with its super compact and efficient Lua web framework and its non-blocking asynchronous sockets. Plus, it comes with everything you need built-in, including an integrated database, SMTP, HTTP client/server, and even IoT protocols like MQTT and industrial protocols like OPC-UA and Modbus.

People

  • Calvin Rose -- bakpakin.com (creator of Fennel lang, moonmint web framework, etc.,)

Compiling Lua source files

Let's say you have a lua program -- hello.lua.

print("Hello Lua!")

The source code can be compiled into a binary file using luac

$ luac -o hello.luac hello.lua

The file hello.luac can then be used in place of the source file. Using the binary file has some advantages like: faster loading, simple source hiding etc.,

To run a .luac file (this is my own convention, the Lua docs do not seem to prefer any particular suffix for the compiled file), you still need to call the lua interpreter,

$ lua hello.luac

Creating standalone executables for lua

A Lua source file can be converted into a executable using different methods. srlua appears to be the popular one.

Download srlua from here and build it using make. Copy glue and srlua files into a convenient location, say ~/bin.

You can now create standalone executable from the Lua source file --hello.lua thus:

$ ~/bin/glue ~/bin/srlua hello.lua hello
$ chmod +x hello
$ ./hello
Hello Lua!

Installing new software packages

Luarocks is the deployment and package management system for Lua. Think CPAN for Perl and PyPI for Python.

I have installed Lua in my $HOME directory. That is, lua interpreter resides in $HOME/bin, include files in $HOME/include etc.,

So, to install Luarocks:

$ export PREFIX=$HOME
$ ./configure --prefix=$PREFIX --sysconfdir=$PREFIX/luarocks --force-config

Installing new lua packages is as simple as:

$ cd
luarocks install luasocket

where luasocket is a new package that is fetched from the luarocks mirrors and installed.

To make this package available to the Lua interpreter export LUA_PATH thusly:

export LUA_PATH="/opt/local/share/lua/5.1//?.lua;/opt/local/lib/luarocks/?.lua;;"

Another LUA_PATH that makes more sense:

?;?.lua;/usr/local/lua/?/?.lua

Also, to explicitly add paths to a script:

# cat t_paths.lua
package.path = package.path.. ";/opt/local/share/lua/5.1/?.lua"
package.cpath = package.cpath.. ";/opt/local/lib/lua/5.1/?.so"
local s = require "socket"
print(s._VERSION)

Incidentally, on the day I was learning aobut luarocks, luaforge.net, the main lua modules repository was down. Friendly folks on the #lua IRC channel told me to add mirrors to the luarocks configuration.

$ vim ~/luarocks/config.lua

I replaced the luaforge.net line with

http://www.luarocks.org/repositories/rocks/",

Misc. Notes

Setting up LUA_PATH and LUA_CPATH is always more complicated than it needs to be. Remember not to have any spaces after the path seperator ;.

LuaDist appears promising to set up virtualenv like sandboxed environments (and more..) for lua.

Lua application packaging

luastatic README tells us that it can package lua applications into executables for windows, macos & linux. A few applications that use luastatic to do their packaging are given. It will be nice to find build setups which can compile to all the three OSes using github CI pipeline.

--

See also: Luarocks.

notes#lua#programming-language

Shell programming

notes#programming-language

Janet

Janet Programming Language is a Clojure like Lisp scripting language also suitable for embedding.

Why

Why Janet?; Apr 2023. Book "Janet for Mortals" by Ian Henry.

Learn

Notes

  • Project definitions go into a project.janet file.
  • Third party dependencies are defined like this: {:dependencies [{:repo "https://github.com/joy-framework/moondown" :tag "0.1.0"}]}

The structure of a project.janet file is: (See more here)

(declare-project
  :name "mylib" # required
  :description "a library that does things" # some example metadata.

  # Optional urls to git repositories that contain required artifacts.
  :dependencies ["https://github.com/janet-lang.json.git"])
  • See littleserver on how to build an executable from Janet source.

Cool projects in Janet

  • Bauble studio -- making art online with lisp(janet) and math.

  • Mendoza -- static website generator with a markdown language inspired by Scribble (of racket).

  • Tester -- testing framework. Readme file has an example of how to use fswatch to run the tests when files change.

  • Joy Framework -- a web framework for Janet.

    • jpm install joy.
    • has an ORM?

  • esv a web application built using Janet, (and circlet library).

Janet Shell

janetsh "A powerful new shell that uses the janet programming language for both the implementation and repl.". This is pretty cool when you want a regular shell that can do mundane things like ls -lart, but also use janet as a REPL and scripting language when you want to more advaned things without resolving to bash scripting. See this introductory article by the creator -- Andrew Chambers.

In this article, -- JSON in Janet Shell you can see how to use the ~/.janetsh.rc file to load additional modules (in this case ability to process JSON) to extend the capability of janetsh.

# Tell janet it can load janet and native modules from ~/janet-modules
(array/insert module/paths 0
   [(string (os/getenv "HOME") "/janet-modules/:all:.janet") :source]
   [(string (os/getenv "HOME") "/janet-modules/:all:.:native:") :native])

# copy combined dynamic library to janet-modules dir
cp build/json.so ~/janetsh-modules

# use it
$ (import json)
$ (var astronaut-in-space
    (json/decode ($$ curl http://api.open-notify.org/astros.json)))
# note: $$ appears to be the function call to invoke programs

# Print each one and the space craft they are in.
$ (each astronaut (astronauts-in-space "people")
    (print
      "Name: " (astronaut "name")
      ", Craft: " (astronaut "craft")))
...
Name: Oleg Kononenko, Craft: ISS
Name: David Saint-Jacques, Craft: ISS
Name: Anne McClain, Craft: ISS

This is a very nice capability (being able to extend the shell programming language).

notes#clojure#lisp#programming-language

Fennel programming language

Fennel is another clojure-like language that compiles down to lua created by Phil HagelBerg "technomancy".

Anywhere you can run Lua code, you can run Fennel code.

Phil Hagelberg/Technomancy has three.blog.posts about using Fennel to write a new game EXO_encounter that is hosted on itch.io (which itself is written by Lua-u* leafo).

Here is an introductory post on Fennel by Technomancy.

Software written using Fennel

*What's the demonym for a Lua programmer? (like Pythonista for Python programmers) -- Lua'r, Lua-oo?

See also: janet, lua

notes#clojure#lisp#lua#programming-language

Mahayana / Vajrayana Practice

tricycle.org

the sutras mention the four reliances:
Do not rely on individuals, rely on the teachings. Do not rely on the
words, rely on the meaning. Do not rely on the adapted meaning, rely on the ultimate meaning. Do not rely on intellectual knowledge_ wisdom.

tricycle.orgPinboardPermalink

#buddhism

C Programming Langauge

Why C?

"The Endurance of an Unmanageable Language" - Some Were Meant for C; via ATS As a Systems Programming Language.

Learning

Tooling

Why not C?

C Is Not a Low-level Language - ACM Queue

Books

People and blogs

Important

  • cdecl; C gibberish ↔ English
  • ~sircmpwn/cstyle -- style guidelines on the lines of "SHOULD" and "SHOULD NOT"s.

C libraries

Memory Management

C programming practices

Cool stuff

Programming Language Theory

we argue that the C standard does not allow Turing complete implementations, and that its evaluation semantics does not preserve typing. Finally, we claim that no strictly conforming programs exist. That is, there is no C program for which the standard can guarantee that it will not crash.

Discussions

Safe C

A Tour of Xr0 - Xr0

Xr0 is a new verifier for the C programming language that eliminates many classes of undefined behaviour, such as use-after-frees (of which double frees are a special case), dangling and null pointer dereferences, memory leaks and the use of uninitialised memory. Most importantly, we believe Xr0 is on a clear trajectory towards enabling programming in C with no undefined behaviour whatever.

notes#c#programming-language

Functional programming

  • (Hughes 1984); link.

Papers

Learning Functional Programming

Book reviews and suggestions

Videos

Misc

-- Xavier Leroy via

References

  • Hughes (1984). Why Functional Programming Matters.
  • Okasaki (1998). Purely Functional Data Structures.

notes#functional-programming#haskell#ocaml

Probability Library

Dr Harry Crane asked for recommendations for a "Probability Library".

These are some of the recommendations:

Harry Crane:

  1. Technical: An Introduction to Probability Theory and Its Applications, Vol. 1, 3rd Edition
  2. Classic: Foundations of the Theory of Probability: Second English Edition (Dover Books on Mathematics) by Kolmogorov
  3. Real World: Antifragile. NNTaleb recommended Statistical Consequences of Fat Tails (book in the works) instead over Antifragile.
  4. Application: Beyond Counting : Exploiting Casino Games from Blackjack to Video Poker by James Grosjean.
  5. Biography: A man for all ages by Ed Thorpe.

Nassim Taleb:

  1. Probability Theory I, 4th Edition and Vol 2 by Loeve
  2. Methods of Mathematical Finance by Karatzas and Shreve
  3. Elements of Information Theory 2nd Edition by Cover and Thomas
  4. Probability and measure, 3rd edition by Patrick Billingsley
  5. Chance and Stability, Stable Distributions and Their Applications by Uchaikin, Vladimir V. / Zolotarev, Vladimir M. PDF
  6. Stable Non-Gaussian Random Processes: Stochastic Models with Infinite Variance by Gennady Samorodnitsky, M.S. Taqqu
  7. Stochastic Processes by J. L. Doob
  8. Limit distributions for sums of independent random variables by B.V.Gnedenko and A.N.Kolmogorov.
  9. Extreme Value Theory by Laurens de Haan and Ana Ferreira
  10. Extreme Value Theory (PDF) by Embrechts et. al.

Misc:

via <https://x.com/BristOliver)

  • Feller vol 1
  • Kolmogorov/Gnedenko
  • Petrov Sums of Independent RVs
  • Grimmett Percolation
  • David Williams Martingales

notes

Python Programming Language

My Python Stuff

Meta

Advanced Python

dabeaz-course/python-mastery: Advanced Python Mastery (course by David Beazley)

Books

  • Fundamentals of Python programming (PDF) -- "... introduction to computer programming textbooks are available in PDF form for free download. University and high school students and self learners around the world have found them helpful. See the terms of use in the preface of each book."

  • Intermediate Python -- "Fluent python" by Luciano Ramalho.

Learning sources

"This repository contains the text for the Tiny Python 3.6 Notebook. it is a notebook containing curated examples for Python 3 as well as the new features found in Python 3.6."

Internals

Developer Experience (DevEx)

Make Python DevEx -- this is very close to how I choose to do Python development. good read. I don't have such an extensive Makefile setup.

Debugging

Profiling

Web

Multiprocessing

Async

Sublime Text

Alternate Python implementations

To Install Zippy:

hg clone https://bitbucket.org/ssllab/zippy
cd zippy
TBD

Libraries

Data science libraries

Conferences

Python3

Pypy

Package management

python setup.py sdist
python setup.py bdist_wheel
twine upload --repository-url https://test.pypi.org/legacy/ dist/*

Project Managers

Do not use requirements.txt - quanttype -- instead choose Poetry, Hatch or pdm.

Poetry

I have been using poetry for a while.

Poetry uses the pyproject.toml standard, but it does not follow the standard specifying how metadata should be represented in a pyproject.toml file (PEP 621), instead using a custom [tool.poetry] table. This is partly because Poetry came out before PEP 621. -- from pdm's documentation

Hatch

pdm

Unlike Poetry and Hatch, PDM is not limited to a specific build backend; users have the freedom to choose any build backend they prefer.

Links

2019

Via My Python Development Environment, 2020 Edition | Jacob Kaplan-Moss:

notes#programming-language#python

Futhark

Futhark is a small programming language designed to be compiled to efficient parallel code. It is a statically typed, data-parallel, and purely functional array language in the ML family, and comes with a heavily optimising ahead-of-time compiler that presently generates GPU code via CUDA and OpenCL, although the language itself is hardware-agnostic and can also run on multicore CPUs. As a simple example, this function computes the average of an array of 64-bit floating-point numbers:

let average (xs: []f64) = reduce (+) 0.0 xs / r64 (length xs)

Futhark is not designed for graphics programming, but instead uses the compute power of the GPU to accelerate data-parallel array computations.

Code in Futhark

notes#programming-language

Algorithms

Algorithm Visualizer

Success stories

Course Pages with Lecture notes

Duke University's CPS130 --Introduction to Design and analysis of Algorithms is an undergraduate course. The lecture notes are good for revising basic concepts.

Jeff Erickson's Algorithm course materials --this is a huge collection of lecture notes, homeworks and exams all in one big document (800+) pages. The tone is conversational, which helps to get a feel of the langauge of the subject. This is a graduate level course.

MIT's Mathematics for Computer Science -- 6.042J, has exhaustive notes on Discrete Mathematics -- 2010

Stanford CS 168: The Modern Algorithmic Toolbox, Spring 2023 by Gregory Valiant; see Tim Roughgarden's Lecture Notes

Cambridge - Algorithms – Course materials

Video Lectures

Articles

Interviews

Books

Free

  • Algorithms by Vazirani et. al
  • Lisp, the Universe and Everything: "Programming Algorithms" Book Work in progress (as of Jan 2020). All the algorithms and data structures are explained using Lisp.
  • Algorithms and Data Structures - The Basic Toolbox by Kurt Mehlhorn and Peter Sanders. [PDF]
  • Algorithms for Modern Hardware - Algorithmica -- Its intended audience is everyone from performance engineers and practical algorithm researchers to undergraduate computer science students who have just finished an advanced algorithms course and want to learn more practical ways to speed up a program than by going from O(nlog⁡n)O(nlogn) to O(nlog⁡log⁡n)O(nloglogn).
  • (Tuzhilin and Zhang 2023) PDF This book is result of Sino-Russian collaboration. It collects the lectures which were given to students of mathematics departments of Moscow State University and Peking University about graph theory and its applications. The book’s narrative is sequential: earlier theorems and definitions are used in later material.

Non-Free

  • Introduction to Algorithms aka CLRS (affl.) -- the standard textbook for this course in large percentage of undergraduate and graduate classes. The book weighs a ton and looks intimidating, but is quite approachable in practice.

Topics

Preliminaries

Probability and Randomness

Dynamic Programming

Divide and conquer algorithms partition the problem into disjoint subproblems, sovle the subproblems recurisvely and then combine their solutions to solve the original problem.

Dynamic programming algorithms apply when the subproblems overlap --ie., when subproblems share subproblems. Dynamic program solves each subproblem just once and saves the answer, thereby avoiding the recomputing the answer.

DP is typically applied to optimisation problems.

Four steps of DP algorithm:

  1. Characterise the structure of an optimal solution.
  2. Recursively define the value of an optimal solution
  3. COmpute the value of the optimal solution, typicall bottom-up.
  4. Construct an optimal solution from computed info.

//Characterise the structure. Define the Value. Compute the Value. Construct.

To Read

2013-11-01: Towards a discipline of experimental algorithms by Bernard Moret.

2013-11-01: O-notation considered harmful (use Analytic Combinatorics instead)

References

notes#algorithms#books#computer-science#data-structures

Ergodicity

Lecture notes – Ergodicity Economics

These notes are our attempt to re-develop economic theory from scratch, namely starting with the axiom that individuals optimize what happens to them over time, not what happens to them on average in a collection of parallel worlds. The latter, surprisingly, is the starting point of the currently dominant form of economic theory. -- Ole Peters

Everything We’ve Learned About Modern Economic Theory Is Wrong - Bloomberg article -- Dec 11, 2020.

(1) How you will go bust on a favorable bet. (Kelly/Shannon/Thorp) - YouTube -- Taleb's video about the

notes

2019