here you goThings I’ve made, read and saved — shared here.

Archive

9,632 entries: 60 from X, 8,925 from Pinboard, saved 1998 to 2026.

Browse by tag, site, author, folder or language

Tags

Sites

Authors

Folders

Languages

Showing 501–547 of 547Folder notes ×

2019

Ballerina programming language

"Ballerina" Could Become the Programming Language of Integration - By dhanushka

Ballerina is a static typed language.

Another special type of variable is that “anydata” data type. This type is a union of the ()|boolean|int|float|decimal|string|(anydata|error)[]|map<anydata|error>|xml|table data types. anydata variables can be used in places where you expect pure values.

Since Ballerina specifically designed to construct apps operating on networks, it natively supports JSON and XML.

json user = {
         fname: "Peter",
         lname: "Stallone",
         "age": age,
         address: {
             line: "20 Palm Grove",
             city: "Colombo 03",
             country: "Sri Lanka"
         }
    };
io:println(user.address.country);
xml x1 = xml `<book>The Lost World</book>`;
io:println(x1);

News

Articles

James Clark's blog:

notes#ballerina#jvm#programming-language

Scala programming language

See Also: Scala for Datascience, ScalaJS, coursier, select-scala

July, 2015 -- Scala book reviews Feb, 2018 -- Recursion and Trampolines in Scala

Random Notes

scala-cli—power repl -S 2.13 -A —ammonite-ver 2.5.11

Notes from Programming in Scala

Chapter 7 - built-in control structures.

  • has very few control structures
  • if while for try match and function calls.
  • scala accumulates control structs in libraries.
  • use val instead of var wherever possible. makes it easy to refactor. "equational reasoning".
  • no = sign while defining a function => returns a Unit value ()
  • while and do-while are NOT expressions. they return Unit.
  • prefer NOT to use while.
  • for
  • "generator" syntax -- file <- files. works on all collections.
  • file is a val.
  • Ranges -- 1 to 4 (yields 1,2,3,4)
  • 1 until 4 -- exlcude the upper bound (yields 1,2,3)
  • generators can also include filters -- for (f <- files if f.getName.endsWith('.py'))
  • nested iteration.
  • mid-stream variable bindings
  • new collection:
def pyFiles =
  for {
    file <- files
    if file.getName.endsWith('.py')
  } yield file
  • IMP: don't put yield inside the curlies.

Exceptions:

try {


} catch {
  case ex: FileNotFoundException => //handle
  case ex: IOException => //handle
}
  • like other control structures, try-catch-finally also results in a value. (this is unlike Java) //imp

Match:


val fruit =
    someArg match {
      case "foo" => "bar"
      case "banana" => "pepper"
      case _ => "what?"
}

Chapter 8- Functions and Closures

First class functions

(x: Int) => x+1

named function:

var increase = (x: Int) => x + 1
  • target typing: someNumbersList.filter((x) => x > 0). the type is inferred from the the list of integers.
  • alternatively: someNumbersList.filter(x => x > 0)

Placeholder syntax:

someNumbersList.filter( _ > 0)
val f = (_: Int) + (_: Int)

Chapter 12 -- Traits

Traits are like java interfaces with concrete methods + declare fields and maintain state.

Traits can do everything classes can do, except --

  1. trait cannot have any clas parameters (ie., those passed to primary constructors).
  2. super calls are not statically bound.

Unlike Java, adding a concrete method to scala trait is an one-time effort.

Chapter 15 -- Case classes and pattern matching


Notes from "Scala in Action"

Chapter 2 -- Getting started

// working with arrays and Lists
@ val nList = 1 :: 2 :: 3 :: 4 :: Nil
nList: List[Int] = List(1, 2, 3, 4)
@ nList.fil
filter     filterNot
@ nList.filter( _ > 2)
res17: List[Int] = List(3, 4)

@ // for comprehensions
@ oldList
res19: List[Int] = List(1, 2)
@ newList
res20: List[Int] = List(1, 2, 3)
@ for { a <- oldList; b <- newList } println (a+b)
2
3
4
3
4
5
@ // above is imperative form
@ // The functional form for comprehension is:
@ // also called the sequence composition
@ for { a <- oldList; b <- newList } yield a+b
res22: List[Int] = List(2, 3, 4, 3, 4, 5)

@ // Pattern Matching
@ val x = 3
x: Int = 3
@ def ordinal(number: Int):  Unit = number match {
    case 0 => println("One")
    case 1 => println("Two")
    case 2 => println("Three")
    case 3 => println("Four")
    case _ => println("what is this?")
  }
defined function ordinal
@ ordinal(x)
Four

@ // Pattern Matching Types
@ dBef printType(obj: AnyRef) = obj match {
    case s: String => println("This is a string")
    case i: java.util.Date => println("This is a Date")
    case l: List[_] => println("An array eh?")
    case _ => println("dunno who ye are")
  }
defined function printType
@ printType(List(1,1,2))
An array eh?


// Guards
@ def numCatcher(num: Int): Unit = num match {
    case x if x < 10 => println("Number less than 10")
    case y if (y%2 == 0) => println("Is an even number")
  }
defined function numCatcher
@ numCatcher(5)
Number less than 10

@ numCatcher(12)
Is an even number

Miscellaneous notes

sbt plays an important role in the day to day programming of scala. I believe 'sbt' stood for "scala build tool" at one time, but is no longer.

To start working on a scala program, you can start with a simple text editor, however using something like IntelliJ IDEA is probably far more productive.

Creating a "project structure" can be automated. Essentially this boilBs down to creating the build.sbt file, project directory and optional plugins.sbt file inside it, a scala file inside the proper src/main/scala/com/example/projectname/ directory etc., There is little reason to do all this manually, except when you are trying to understand the reason behind such a recommended directory layout.

giter8 ie., the g8 is a tool that can create a project structure based on pre-defined templates. Think of this as paster tool for Python.

When you have g8 installed, creating a new project (say, a basic scala project using this template ), all you have to do is:

$ g8 fayimora/basic-scala-project.g8

A basic scala project

name [Basic Project]: basic-scala
organization [com.example]: com.btbytes
version [0.0.1]:

Template applied in ./basic-scala

$ cd basic-scala
$ ls
build-sbt src

$ sbt run will download the dependencies defined in the build.sbt file,B compile source and run the program.

sbt can also create the files necessary to make this an "IntelliJ IDEA project". For this, we add a dependency to the plugins.sbt file:

#project/plugins.sbt -- add this line
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.6.0")

Running sbt gen-idea will generate the IDEA project files (.idea, some .iml).

Launch up IDEA and "open project" and open the root directory of the project. You are now set to work on the code.

Packaging the binary to a .jar:

$ sbt packageBin

Will produce target/scala-2.10/basic-scala_2.10-0.0.1.jar (where basic-scala is the name of the project).

Why not Scala

Concepts

Covariance, Contravariance and Invariance

Reading:

In general, covariant type parameter can be used as immutable field type, method return type and also as method argument type if the method argument type has a lower bound. Because of those restrictions, covariance is most commonly used in producers (types that return something) and immutable types.

Adhoc polymorphism

In this article we discussed the motivation for introducing ad hoc polymorphism and implementing it using the Type Class Pattern in Scala. It allows us to build abstractions which are completely decoupled from the underlying data types on which they operate. Hence we can implement polymorphic functions which operate on types we have no control over, such as types defined in the standard library or some other third party library, without sacrificing the static type safety.

Concurrency (Akka etc.,)

StBreams

Libraries

Tools

  • lihaoyi/Ammonite: Scala Scripting. Ammonite enables shell-like scripting in the Scala programming language.
  • Conscript -- is a tool for installing and updating Scala software programs.
  • giter8 -- is a command line tool to generate files and directories from templates published on github or any other git repository.

SBT

Ensime -- Enhanced Scala

Interaction Mode for Emacs.

Talks

Articles

Blogs

Libraries

  • kantan.csv has good documentation/tutorials.
  • Quill Quoted Domain Specific Language (QDSL) to express queries in Scala and execute them in a target language. The library’s core is designed to support multiple target languages, currently featuring specializations for Structured Query Language (SQL) and Cassandra Query Language (CQL).

REPL

Ammonite REPL is much more featureful than the default REPL and comes with a lot of ergonomic improvements and configurability that may be familiar to people coming from IDEs or other REPLs such as IPython or Zsh.

Links

Scalajs

Scala Native

Scala CLI

With scala-cli, it is quite easy to package a scala program into a standalone program:

scala-cli --power package --native-image Echo.scala -o echo -- --no-fallback

where Echo.scala is:

object Echo {
def main(args: Array[String]): Unit =
	println(args.mkString(" "))
}
$ ls -alrt echo
-rwxr-xr-x@ 1 pradeep  staff  14120984 Aug 23 17:15 echo
$ du -sh echo
 13M    echo

Books

Tutorials

Indyscala notes

Feb 6, 2017 -- Resource management in Scala

Presenter -- Ross A. Baker

May 1, 2017

notes#jvm#programming-language#scala

Mercury Programming Language

Mercury language -- Home page

Mercury combines (predicate) logic programming and one of the more rigorous type systems to give a programming language that is declarative, efficient and consistent.

-- Good explanation of Mercury syntax by Markus Triska, a prolog programmer; Apr 2019.

notes#programming-language

Ada Programming Language

Why Ada

Blogs

Ada learning resources

Ada Code examples

Ada Editors

Ada Books

  • Ada for Software Engineers
  • Concurrent and Real-Time Programming in Ada

Ada resources on the net

Ada articles

I don't know if Zen of Python was directly influenced by the Steelman document, or it's a result of convergent evolution, but they bear a striking similarity in their spirit. Perhaps it's simply because they should be requirements for any well designed language!

notes

Kubernetes

Articles

Related software

Learn

I had a similar experience myself with Kubernetes. The first time I tried to learn it, it was a bewildering morass of jargon—all those namespaces and containers and Pods and Deployments and Services and Ingresses just to get a simple HTTP server running! Then I read a networking textbook and everything made much more sense. The (arguably) most complicated parts of Kubernetes exist to solve networking-related problems—allowing hundreds of containers to talk to each other independently while hosted on a much smaller set of computers—so the networking textbook gave me a schema onto which I could hang all my Kubernetes factoids. Once I knew how Linux’s IP routing, iptables, and network namespaces worked, it was much easier for me to understand what exactly something like “kube-proxy” was doing. -- In defense of blub studies

News

A better Kubernetes, from the ground up · blog.dave.tf; Nov 2020

Tools

notes

Splunk

Remove indexes and indexed data:

splunk stop
splunk clean eventdata -index _internal -f

See search history - |history

See also:

notes

GNU/Linux, MacOS

See also:

Keep in mind to explore:

Notes

  • To convert the .pub file into pem format -- ssh-keygen -f rsa.pub -e -m pem
  • pidof <appname> returns the pid of the application. useful for killing the app etc..
  • The sock file for mysql5 installed using macports lives at /opt/local/var/run/mysql5/mysqld.sock . You will need this when using thirdy party SQL clients such as Seqel Pro
  • Use python_select to select the default python that is used on your mac.
  • If a download is corrupted while installing from macports, try sudo port clean --all <the offending package>
  • Upgrade ports and all other software: sudo port selfupdate and sudo port upgrade outdated.
  • X environment tricks for Mac OS X.

Pick up *NIX skills

Installing OpenJDK on Mac OSX

  • Download Soylatte

  • Unzip to ~

  • Edit ~/.bashrc and add these lines:

     JAVA_HOME=~/java
     export JAVA_HOME
     export PATH=~/java/bin:$PATH
     
  • activate changes with $source ~/.bashrc

Burning ISO images to CD/DVD from shell

DVD:

$ growisofs -dvd-compat -Z /dev/hdc=dvd.iso

CD: Find out the cd recording device using cdrecord -scanbus

$ $ cdrecord -scanbus
scsibus2:
        2,0,0   200) 'HL-DT-ST' 'DVDRAM GH22NS50 ' 'TN01' Removable CD-ROM

Use cdrecord to burn.

$ cdrecord -v -dao dev=2,0,0 speed=8 file.iso

don't want namservers handed out by local DHCP server?

Comment out domain-name-servers, in /etc/dhcp/dhclient.conf.

Tell apt to add new pgp key to its list

$ curl -s http://someurl/key.pgp | sudo apt-key add -
$ sudo apt-key list

Label Drives using e2label command

$ sudo e2label /dev/sdg1 emdebian

Compiling stuff in 32 bit mode on Mac OSX 10.6

Add these flags to ./configure.

CFLAGS="-O -arch i386" CXXFLAGS="-O -arch i386" LDFLAGS="-arch i386"

Produce print ready copies of man docs

$ man -t `man -w tmux` | ps2pdf - tmux.pdf

Change keyboard layout in the shell

$ setxkbmap us

2014/11/25: swapped caps and left-ctrl. Put line below in /etc/default/keyboard

XKBOPTIONS="lv3:ralt_alt,compose:menu,ctrl:nocaps"

and run sudo dpkg-reconfigure keyboard-configuration and confirm changes to keep the changes you have made.

find memory usage by a process(?)

$ ps -Ao pid,rsz,args | grep <name>

src

Enabling anti-aliased fonts on Ubuntu

Edit $IDEA/bin/idea.vmoptions file and add these options:

-Dswing.aatext=true
-Dawt.useSystemAAFontSettings=lcd

The above options managed to get smooth font rendering on Ubuntu 12.10 using OpenJDK7.

Resources

OpenBSD

Minix

Debian

Measuring stuff

  • osquery -- allows you to easily ask questions about your Linux and OSX infrastructure. Whether your goal is intrusion detection, infrastructure reliability, or compliance.

Running log of things

notes

Computer Science

A jumping off point for cs topics.

Curriculum

Concepts

  • CS for all -- "In a nutshell, our objective is to provide an introduction to computer science as an intellectually rich and vibrant field rather than focusing exclusively on computer programming. While programming is certainly an important and pervasive element of our approach, we emphasize concepts and problem-solving over syntax and programming language features." At Mudd, this course is taken by almost every first-year student—irrespective of the student’s ultimate major—as part of our core curriculum.

  • Introduction to Theoretical Computer Science "This is a textbook in preparation for an introductory undergraduate course on theoretical computer science. I am using this text for Harvard CS 121."

  • The "Computer Science Field Guide" is an online interactive resource for high school students learning about computer science, developed at the University of Canterbury in New Zealand.

  • Graph Coloring Methods

  • Great Ideas in Theoritical Computer Science -- CS251 at CMU.

Constructing proofs in the context of CS

via this twitter discussion

Reading Lists

Forgotten ideas of Computer science

Joe Armstrong, the creator of Erlang programming language, asked on twitter (2018/1/11):

: "I’m interested in the forgotten ideas of computer science. Needed for a talk.Can you post examples of great CS ideas that have been largely forgotten. Examples: Linda tuple spaces, Boyer-Moore algorithm". Fascinating response.

ChanningWalton:

: Linda tuple spaces (cf: Fly object space by @NigeWarren, gigaspaces's commercial offering). "What are tuple spaces and Linda tuple spaces good for?" One use is the blackboard idea. Tuples are put into the space, processes observing the space can be notified of interesting tuples, ‘take’ one or all, perform work and ‘put’ new tuples into the space for other workers to pick up. Tuples can also be leased out from the space, this guarantees only one process can lease a tuple to do something with it. JavaSpaces is based on these ideas. [Argorak: Interesting. Has this influenced the idea of a "document based application" comes from in OS X? Which for example doesn't close when the last window closed, ready for handling others?] Not really, that's how OS 9 works also. The "document-based" thing in OS X is mostly about a way of structuring the OOP in your application to best take advantage of the builtin capabilities of NSDocument. The most OpenDoc-y idea in OS X is the little-used "Services" menu. Oddly OpenDoc DID massively influence Microsoft, who stripped down the basic concept to a less-ambitious, but much more practical, set of ideas around IPC/software components. Their version was called OLE, which became COM, which set the basic design of C#/.NET.

enkiv2:

: Transpointing windows Regex derivatives Enfilades Cord routing

: Specific tech: 9p, MUMPS, XU88, network-transparent object method forwarding, icon/unicon, APL, bitmapped keyboards, literal cranks for debugging, distributed expert systems (like the 5th generation computer project)

: Plan9, Xanadu, colorforth, the legacy Amiga community, & the Squeak Smalltalk community each have a greater number of interesting ideas than the whole of the web dev & devops world combined, and all those ideas are 30+ years old.

Hillelogram:

: APLs and special purpose langauges, Program Synthesis For Declarative Building Design

thompson_si:

: - Lucid - coroutines

rtraschke:

: - Croquet TeaTime protocol for synchronised distributed objects

  • Arena based memory allocation
  • Using make to describe artefact dependencies, instead of as a programming language

robertrodermann:

: Inmos Transputer, Occam. π-calculus

mcclure111:

: OpenDoc -- Briefly in the mid-90s Apple and IBM collaborated on a tech that inverted the idea of a gui "application" and refocused around "documents" as the core idea. You'd be working on a piece of data, and software would exist sort of in a cloud around your document things you previously wrote as an application would be broken down into a cloud of individual features. Features could be mixed and matched from different apps, so you could be in Word using Wordperfect's spellchecker and editing one of the graphics using Photoshop's tools.

@andrew_j_stone:

: E (programming language) (justincormack: Still alive in capnproto rpc! Directly based on E https://capnproto.org/rpc.html) (evanplaice: The capabilities sound eerily like modern JS + AJAX + Rxjs/promises/async/await. Specifically, the event-callback model to avoid deadlocking, fetchi g distributed resources asynchronously, and using promises to ensure order of execution on async code.)(KentonVarda: The concept of a Promise in both Cap'n Proto and JavaScript is derived from E's promises, hence they all look similar. However, "capabilities" are an entirely separate concept, not directly related to promises or async event loops. On the Cap'n Proto RPC page, scroll down to "distributed objects" -- that's the part about capabilities.)

jcoplien:

: 1s complement arithmetic. Lissajou status trace console displays with the upper 8 address bits driving the X-axis and the lower 8 bits driving the Y-axis. Twister-card memory. BCD ALUs. ABACABAD scheduling. Leaky bucket counters. Coroutines. Buddy system. Use of XOR'ed links to traverse a list in either direction with just one link field. And, to be snarky, object-oriented programming, forgotten in about 1980 and replaced by class-oriented programming.

jminguillona:

: Bresenham's algorithm for drawing lines and other graphic primitives

GabeMoralesVR:

: More on graphics side of things, but deferred tile-based rendering. Only PVR really explored it, but it makes so much more sense than traditional rasterization. Free z-culling, essentially infinite fill rate, no wasted rasterization. Sadly, apple ditched PVR, so likely dead tech.

exclamatio:

: MASCOT, DARTS, all those '80s real-time system methodologies

Kitten_tech:

: The NetBLT protocol: net flow control/retransmission based on transferring blocks, rather than a stream like TCP.

DavidTWeaver:

: - The Marr Hypothesis

  • Multi-Agent Systems
  • Boids Algorithm
  • Cybernetics (or at least, it doesn't get emphasized in CS anymore)
  • Fuzzy Logic

mhollemans:

: - self modifying code

plragde:

: Treaps. Brzozowski derivatives. FKS perfect hashing. (TommyEttinger: Brzozowski derivatives and their successors in partial derivatives are still actively studied in some circles, though I think they have a ways to go to catch highly-optimized traditional regex libraries for efficiency. Great potential though.) (ysr1729: They have been used in decision problems in logic and computer science -- e.g. for satisfiability of interval temporal logic using automat-theoretic methods.)

Ngnghm:

: Orthogonal Persistence. Reflection. Fexprs. Lisp Machines. The Systems Paradigm. FORTH. The TUNES project. Having lots of simple connected CPUs with each unit of RAM, instead of a overly complex and securitywise hazardous central CPU behemoth (connection machine, greenarrays). (phenidone: Persistence: true and that makes me a little sad - my PhD on GC was implemented in an orthogonal-persistence storage layer. Reflection is still a big thing at least in Java-land, and there is a bit of a forth fanboy-revival in the microcontroller space.)

madrawls:

: clockless flow computing

danbjson:

: Analogous machines, ie solving a diffential differential equation by building an electric circuit with the same characteristics and then measuring the current over time.

KalmanReti:

: Pattern matching as a primitive operation in a language SNOBOL

hypercubed:

: (3) Stanford Seminar - Concatenative Programming: From Ivory to Metal - YouTube

notes

Lists

The best facts I heard this year (2025) - Zhengdong Wang; list of facts overheard.

Agnes Callard's 51 tips, via Zhengdong Wang


Prompt: What lists do you keep?; My attempts at thinking what could be good^Hfun lists to keep:

  • books to read
  • authors to read
  • places to visit
  • temples to visit
  • shlokas and mantras to memorize
  • list of opening phrases to break ice
  • list of questions to get people to talk to you/know them.
  • list of songs to sing
    • solo
    • in a grownup group
    • in a childrens group
    • devotional
    • in different languages
  • newsletters that I subscribe to
  • Talks I liked


See also: reading-lists

notes

Racket Programming Language

Libraries

  • Video Language -- "Video is a language for making movies. It combines the power of a traditional video editor with the capabilities of a full programming language."

Articles

Code

Exciting

Videos

My notes

  • Racket cannot pin version dependencies yet. Does common-lisp have the same issue? Is it a Lispism?

notes#programming-language

2018

Logic Programming

To Read:

  • Sterling and Shapiro (Sterling and Shapiro 1986) is often recommended as the introductory text to Logic Programming.
  • Nominal Logical Programming (Cheney and Urban 2008) -- is an extension of first-order logic which provides a simple foundation for formalizing and reasoning about abstract syntax modulo consistent renaming of bound names (that is, alpha-equivalence). "powerful tool for informal reasoning about theoretical computer science. For me it made theoretical computer science tangible, and I hope these series of posts can do the same for the reader." -- swannodette
  • miniKanren -- is an embedded Domain Specific Language for logic programming. See also: Will Byrd's Thesis -- Relational Programming in miniKanren: Techniques, applications and implementations (Byrd 2010).
  • For real world applications of logic programming see -- Concepts, Techniques, and Models of Computer Programming by Roy, Haridi and Seif (Van Roy and Haridi 2004).

Software

  • logic-db is a js lib to combine logic programming and database

Misc

References

  • Sterling, Shapiro (1986). The Art of Prolog: Advanced Programming Techniques.
  • Cheney, Urban (2008). Nominal Logic Programming.
  • Byrd (2010). Relational Programming in Minikanren: Techniques, Applications, and Implementations.
  • Van Roy, Haridi (2004). Concepts, Techniques, and Models of Computer Programming.

notes#programming-language

Probabilistic Programming

"What is the TL;DR of Probabilistic programming" (via HN)

Reading List – MIT Probabilistic Computing Project

What is probabilistic programming? - O'Reilly Media. So It Begins: Darpa Sets Out to Make Computers That Can Teach Themselves | WIRED

Probabilistic programming language - Wikipedia

Practical Probabilistic Programming by Avi Pfeffer who also created the Figaro language [PDF].

Probabilistic Programming Wiki

[1809.10756] An Introduction to Probabilistic Programming by Jan-Willem van de Meent, Brooks Paige, Hongseok Yang, Frank Wood; a 200+ page tutorial.

Online / programming env

Courses

notes#probability#programming

Literate Programming

Let us change our traditional attitude to the construction of programs: Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to humans what we want the computer to do. -- Donald E. Knuth, Literate Programming, 1984

Examples of literate programming

  • ulix a literate, teaching operating system.
  • knitr Elegant, flexible and fast dynamic report generation with R
  • stich -- is a small python library for writing reproducible reports in markdown. It's heavily inspired by (read: a python clone of) knitr and RMarkdown.
  • ferret.org the entire source of Ferret programming language as an emacs org file.

Tools

Literate -- A modern literate programming system written in D

brew tap zyedidia/literate
brew install --HEAD zyedidia/literate/literate

In addition to those of CWEB:

  • Markdown based -- very easy to read and write Literate source.
  • Generates readable and commented code in the target language (the generated code is usable by others)
  • Reports syntax errors back from the compiler to the right line in the literate source
  • Supports any language including syntax highlighting and pretty printing in HTML
  • Compatible with Vim (literate.vim)
  • Highly customizable (you can add your own HTML or CSS)
  • Runs fast -- wc.lit compiled for me in 7ms for both code and HTML output
  • Supports TeX equations with $ notation.

Inspired from CWEB:

  • Automatically generates hyperlinks between code sections
  • Formatted output similar to CWEB

I wrote lua4py page using Literate.


See also the Nimib project by Nim programmers.

notes#literate-programming

Build Systems

Writing

Build systems

Cloud

Earthly - Make Builds Super Simple; HN; See example use - Improving Testing & Continuous Integration in Phoenix - Phoenix Blog - Jan 2021.

notes

Sqlite

I wrote an article about SQLite for DeveloperIQ magazine published in Jan 2005 -- Move Over MySQL, SQLite is here!!


Query Parquet files in SQLite | Colin Dellow

Importing csv into sqlite: see this page for extended usage

sqlite> .import myfile.csv mytable

SQLite as a document database using Generated Columns and The JSON1 Extension

Tutorials

SQLite Date & Time - How To Handle Date and Time in SQLite (The site sqlitetutorial.net seems to have handy reference for common tasks with sqlite.)

SQLite as a Collaborative & Multiplayer Application File Format - YouTube

Optimizing SQLite for servers

60K RPS out of sqlite on a $5 VPS

Django

Tools around SQLite

sqlpkg

The list of packages are here - https://sqlpkg.org.

#TODO SQLPkg is written in go. install it manually.

Sqlite-utils

can do things like:

  • Running queries and returning JSON, CSV, markdown table
  • Inserting JSON data: `echo | sqlite-utils insert dogs.db dogs -
  • Inserting CSV data: sqlite-utils insert dogs.db dogs docs.csv --csv
  • Insert-replacing data.

sqlite-http

Introducing sqlite-http: A SQLite extension for making HTTP requests / Alex Garcia | Observable

Sqlean

nalgeon/sqlean: The ultimate set of SQLite extensions

  • crypto: hash functions
  • define: user-defined functions and dynamic sql
  • fileio: read and write files
  • fuzzy: fuzzy string matching and phonetics
  • ipaddr: IP address manipulation
  • json1: JSON functions
  • math: math functions
  • regexp: regular expressions
  • stats: math statistics
  • text: string functions
  • unicode: Unicode support
  • uuid: Universally Unique IDentifiers
  • vsv: CSV files as virtual tables

Pocketbase

Pocketbase is an open source backend that can be used to build API applications.

  • single binary download
  • written in Go
  • embedded database (SQLite)
  • built in user and files managment
  • built in admin UI
  • simple REST UI

Sqlsync

orbitinghail/sqlsync SQLSync is a collaborative offline-first wrapper around SQLite. It is designed to synchronize web application state between users, devices, and the edge.

Example use cases

  • A web app with a structured file oriented data model like Figma. Each file could be a SQLSync database, enabling real-time local first collaboration and presence
  • Running SQLSync on the edge with high tolerance for unreliable network conditions
  • Enabling optimistic mutations on SQLite read replicas

Soul

soul | 🕉 A SQLite REST and realtime server, written in node.js; Run soul -d sqlite.db -p 8000 and it’ll start a REST API on http://localhost:8000 and a Websocket server on ws://localhost:8000


See also: duckdb, til/sqlite-writable_schema, code/sqlite-csv-loading-using-virtual-table-extension

notes#database#sqlite

bash scripting

Never forget to run shellcheck on any bash/shell script you write.

Bash for production systems - presentation on “Survival guide for quality scripts”

YJL: bashlint and ShellCheck for checking Bash sources

dylanaraps/pure-bash-bible: 📖 A collection of pure bash alternatives to external processes.

——

Minimal safe Bash script template | Better Dev

gist source:

#!/usr/bin/env bash

set -Eeuo pipefail

cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1

trap cleanup SIGINT SIGTERM ERR EXIT

usage() {
  cat <<EOF
Usage: $(basename "$0") [-h] [-v] [-f] -p param_value arg1 [arg2...]

Script description here.

Available options:

-h, --help      Print this help and exit
-v, --verbose   Print script debug info
-f, --flag      Some flag description
-p, --param     Some param description
EOF
  exit
}

cleanup() {
  trap - SIGINT SIGTERM ERR EXIT
  # script cleanup here
}

setup_colors() {
  if [[ -t 2 ]] && [[ -z "${NO_COLOR-}" ]] && [[ "${TERM-}" != "dumb" ]]; then
    NOCOLOR='\033[0m' RED='\033[0;31m' GREEN='\033[0;32m' ORANGE='\033[0;33m' BLUE='\033[0;34m' PURPLE='\033[0;35m' CYAN='\033[0;36m' YELLOW='\033[1;33m'
  else
    NOCOLOR='' RED='' GREEN='' ORANGE='' BLUE='' PURPLE='' CYAN='' YELLOW=''
  fi
}

msg() {
  echo >&2 -e "${1-}"
}

die() {
  local msg=$1
  local code=${2-1} # default exit status 1
  msg "$msg"
  exit "$code"
}

parse_params() {
  # default values of variables set from params
  flag=0
  param=''

  while :; do
    case "${1-}" in
    -h | --help)
      usage
      ;;
    -v | --verbose)
      set -x
      ;;
    --no-color)
      NO_COLOR=1
      ;;
    -f | --flag) # example flag
      flag=1
      ;;
    -p | --param) # example named parameter
      param="${2-}"
      shift
      ;;
    -?*)
      die "Unknown option: $1"
      ;;
    *)
      break
      ;;
    esac
    shift
  done

  args=("$@")

  # check required params and arguments
  [[ -z "${param-}" ]] && die "Missing required parameter: param"
  [[ ${#args[@]} -eq 0 ]] && die "Missing script arguments"

  return 0
}

parse_params "$@"
setup_colors

# script logic here

msg "${RED}Read parameters:${NOCOLOR}"
msg "- flag: ${flag}"
msg "- param: ${param}"
msg "- arguments: ${args[*]-}"

——

notes#unix

Machine Learning

Learning

When people are getting ready to learn machine learning at MSc level, they don’t learn from a course that says “if you don’t understand it it doesn’t matter”, they learn from this course instead. Abu-Mostafa’s Caltech course is probably the best first course that delivers good understanding of machine learning. via

Resources

Video

Blogs

Papers

Browse the State-of-the-Art in Machine Learning | Papers With Code

Pen and Paper Exercises in Machine Learning

This is a collection of (mostly) pen-and-paper exercises in machine learning. The exercises are on the following topics: linear algebra, optimisation, directed graphical models, undirected graphical models, expressive power of graphical models, factor graphs and message passing, inference for hidden Markov models, model-based learning (including ICA and unnormalised models), sampling and Monte-Carlo integration, and variational inference.


See also: machine-learning-books

notes#learning#machine-learning

Julia programming language

Learning

Blog Posts

Comparisons

2020-06-03 A Look at Chapel, D, and Julia Using Kernel Matrix Calculations – The D Blog; Julia comes out ahead in performance. Even handed comparison on Dlang's website.

Books/Tutorials

IDE/tools

Important information

twitter

notes#julia#programming-language

Programming Langauges Theory

Notes on theory of programming languages.

Concepts See:

Metalists

  • A path to Programming Language Theory enlightenment
  • Colin Stebbins Gordon's Electronic References -- "... loosely-categorized collection of links to PL (and related) texts and references that are available online because they are either openly shared publications (like PLAI, SF, or the HoTT book), first editions of updated books released to the world (like Programming in Scala or Semantics with Applications), thorough but out of print references (like TTFP), or publicly available drafts (like HtDC and Practical Foundations for Programming Languages). Most of these I've only used for brief personal reference, and have not read in depth. The exceptions, those books I've spent considerable time with and highly recommend, are marked with asterisks."

Books

Reading

Papers

Courseware

Lectures

Denotational Semantics

notes

J programming language

J programming language

Code

Blogs

People

notes#apl#jlang#programming-language

CSV

NOTE: When you can, prefer ASV over CSV -- Text File formats – ASCII Delimited Text – Not CSV or TAB delimited text, especially  when you aren’t writing the files by hand. (Caveat: "UTF-8 won’t trip this since it is ASCII compatible. That is, any ASCII byte in valid UTF-8 encoded bytes corresponds to its ASCII codepoint, regardless of context. (All continuation bytes are at least 0x80 and all leading bytes for codepoints greater than U+007F are at least 0xC0, which makes the 0x1E and 0x1Fseparators safe.") UTF-16 or UTF-32 on the other hand...)

Convert CSV to ASV: xsv fmt -t$'\x1f' one.csv > one.asv

Command line tools

MacOS

  • Table Tool is a fast, open source, mac native application to open csv files. I have set this as my default CSV opener.

Web tools

csvbase, a simple web database, faq

  • View, add, edit and delete rows from a browser
  • Export to CSV, XLSX, Parquet, etc
  • Each table has a single URL, RESTfully accessed by HTTP verb
  • has an API with API key, that you get when you register
  • works out of the box with R, pandas, duckdb, apache-spark
  • paste cells directly from Microsoft Excel
  • code is open source on github under AGPL license, written in python.

Apache Metamodel

Apache Metamodel provides a common interface for discovery, exploration of metadata and querying of different types of data sources. Uniform connector and query API for CSV & data formats, relational databases, noSQL databases.

With MetaModel you use a type-safe SQL-like API for querying any datastore:

DataContext dataContext = DataContextFactory.create[TypeOfDatastore](...);
DataSet dataSet = dataContext.query()
    .from("libraries")
    .select("name")
    .where("language").eq("Java")
    .and("enhances_data_access").eq(true)
    .execute();

The MetaModel query API allows you to use the power of SQL, even on data formats such as CSV files, Excel spreadsheets, NoSQL databases and more.

notes#csv#query#tools

Type Theory

Introductory materials

Important concepts in type theory.

  • Alonzo Church's typed lambda-calculus -- mathematical foundation

  • Per Martin-Löf's intuitionistic type theory, 1998 :cite:martin1998intuitionistic

  • Curry-Howard correspondence -- equates programs with proofs.

  • Homotopy type theory refers to a new interpretation of Martin-Löf’s system of intensional, constructive type theory into abstract homotopy theory.

Type inference

Type Inference by Solving Constraints – Caleb Helbling – Programming Languages & Systems

Videos

  • "Type Theory Foundations" lecture by Robert Harper at OPLSS 2013.

To read

  • "Programming in Martin-Lof's type theory" by Nordstrom, 1990 cite:[Nordstrom1990]

Row polymorphism vs subtyping

Misc

notes

ATS Lang

What is ATS?

Learning

Programs written in ATS

Talks

Blog posts

notes#programming-language

Vala Programming Language

STATUS: No longer Maintained.

Vala is a C# | Java like langauge that compiles down to C+glibc. Used heavily for developing gnome desktop applications. Before I discovered D (and to certain extent nim), I had hoped that Vala would be a nice compiled langauge to use. However, the community of Vala appears to not care beyond desktop applications.

notes#programming-language

Type system

Ability to read type rules is an important skill when reading Programming Langauge theory papers.

The two books that have been mentioned in relation to learning type systems are:

  • Types and Programming Languages by Benjamin Pierce
  • Practical Foundations for Programming Languages by Robert Harper

Papers

Discussions

Articles

Courses

notes

Red language

Link dump

Unsorted links ...

Why Red

Learn

Community

Rebol stuff

People / blog / website

Code

notes

Pure programming langauge

Pure is a modern-style functional programming language based on term rewriting. It offers equational definitions with pattern matching, full symbolic rewriting capabilities, dynamic typing, eager and lazy evaluation, lexical closures, built-in list and matrix support and an easy-to-use C interface. The interpreter uses LLVM as a backend to JIT-compile Pure programs to fast native code.

REPL

: 1. offers completion 2. show foo will print the definition of defined functions and vars.

Language notes

: 1. records == "hash pairs"

notes

Programming Aphorisms

Norris Numbers ---

My friend Clift Norris has identified a fundamental constant that I call Norris’ number, the average amount of code an untrained programmer can write before he or she hits a wall. Clift estimates this as 1,500 lines. Beyond that the code becomes so tangled that the author cannot debug or modify it without herculean effort.


Aphorisms on programming language design

notes

Operating Systems

Unikernels

Courses

Books

History

Teaching OSes

  • eduOS-rs · eduOS-rs is a Unix-like computer operating system based on a monolithic architecture for educational purposes, which is developed for the course Operating Systems at the RWTH Aachen Univeristy.

Interesting OSes

notes

NGiNX

See also: openresty

Notes

Use strace to verify whether NGiNX is serving gzipped static assets:

$ sudo strace -p 25249 2>&1  | grep gz

where 25249 is the pid of the nginx worker process, which can be found by ps ax | grep nginx

Refs

Applications

  • Telize is a JSON IP and GeoIP REST API (IP Geolocation) built on Nginx and Lua. via

Configuration tools

notes

ಮೂಡಲ ಮನೆಯ ...

ಚಿತ್ರ: ಬೆಳ್ಳಿ ಮೋಡ, 1967 Youtube video

ಕವಿ: ದ. ರಾ. ಬೇಂದ್ರೆ

ಮೂಡಲ ಮನೆಯ ಮುತ್ತಿನ ನೀರಿನ

ಎರಕಾವಾ ಹೊಯ್ದ, ನನ್ನನೆ ಎರಕಾವಾ ಹೊಯ್ದ

ಬಾಗಿಲ ತೆರೆದು ಬೆಳಕು ಹರಿದು

ಜಗವೆಲ್ಲಾ ತೊಯ್ದ, ದೇವನು ಜಗವೆಲ್ಲಾ ತೊಯ್ದ

ಎಳೆಗಳ ಮೇಲೆ, ಹೂಗಳ ಒಳಗೆ

ಅಮ್ರಿತದಾ ಬಿಂದು, ಕಂಡವು ಅಮ್ರಿತದಾ ಬಿಂದು

ಯಾರಿರಿಸಿಹರು ಮುಗಿಲಿನ ಮೇಲಿಂದ

ಇಲ್ಲಿಗೆ ಇದ ತಂದು, ಈಗ ಇಲ್ಲಿಗೆ ಇದ ತಂದು

ಗಿಡಗಂಟೆಗಳ ಕೊರಳೊಳಗಿಂದ ಹಕ್ಕಿಗಳ ಹಾಡು

ಹೊರಟಿತು ಹಕ್ಕಿಗಳ ಹಾಡು

ಗಂಧರ್ವ ಸೀಮೆಯಾಯಿತು, ಕಾಡಿನ ನಾಡು

ಕ್ಷಣದೊಳು, ಕಾಡಿನ ನಾಡು


Notes:

notes

ಕನ್ನಡ ಪ್ರಾರ್ಥನೆ -- "ಕನ್ನಡವೆಮ್ಮಯ ಪ್ರಾಣ"

(Note: prayer before the Sunday Kannada class)

Kannadavemmaya praaNa, kannadavemmaya maana

ಕನ್ನಡವೆಮ್ಮಯ ಪ್ರಾಣ, ಕನ್ನಡವೆಮ್ಮಯ ಮಾನ

Kannada taayige namana, O kannada taayige namana

ಕನ್ನಡ ತಾಯಿಗೆ ನಮನ, ಓ ಕನ್ನಡ ತಾಯಿಗೆ ನಮನ ।।2।।


Kannada taayiya makkaLu naavu

ಕನ್ನಡ ತಾಯಿಯ ಮಕ್ಕಳು ನಾವು

Kannadigaru naavu ennuta kuNivevu

ಕನ್ನಡಿಗರು ನಾವು ಎನ್ನುತ ಕುಣಿವೆವು ।।2।।


Kannada naDe-nuDi hrudayangamavu

ಕನ್ನಡ ನಡೆ-ನುಡಿ ಹೃದಯಂಗಮವು

KannadatanavE siri-sampadavu

ಕನ್ನಡತನವೇ ಸಿರಿ-ಸಂಪದವು. ।2।


Kannadavemmaya praana, kannadavemmaya maana

ಕನ್ನಡವೆಮ್ಮಯ ಪ್ರಾಣ, ಕನ್ನಡವೆಮ್ಮಯ ಮಾನ

Kannada taayige namana, O kannada taayige namana

ಕನ್ನಡ ತಾಯಿಗೆ ನಮನ, ಓ ಕನ್ನಡ ತಾಯಿಗೆ ನಮನ ।।2।।


baaLali kannada, beLeyali kannada

ಬಾಳಲಿ ಕನ್ನಡ, ಬೆಳೆಯಲಿ ಕನ್ನಡ ।2।

uLiyali kannada endendU

ಉಳಿಯಲಿ ಕನ್ನಡ ಎಂದೆಂದೂ ।2।

kannadatanavE nitya

ಕನ್ನಡತನವೇ ನಿತ್ಯ


KannadatanavE nityavu enutali

ಕನ್ನಡತನವೇ ನಿತ್ಯವು ಎನುತಲಿ

Kannada taayige namisuvevu

ಕನ್ನಡ ತಾಯಿಗೆ ನಮಿಸುವೆವು ।4।


notes#kannada

Indy Big Data meetup - December 2015

Talk by Josh Berkus, core member of PostgreSQL.

Big data features

  • sophisticated sql parser/planner
  • works with most DW tools
  • many large DB features (partitioning etc.,)
  • Foreign Data Wrappers (FDW)
  • stability and reliability

Data sizing:

  • comfort zone: 1G to 10TB
  • possible: 10TB to 100TB
  • With extensions: ??

9.5 status:

  • beta 2 released Nov 12, 2015.

9.5 big data features:

  • UPSERT
  • BRIN indexes
  • GROUPING SETS/CUBE
  • Foreing Schema
  • Faster sorts
  • Misc

UPSERT: INSERT ... ON CONFLICT DO { UPDATE | IGNORE }, similar to MERGE only better. Merge syntax was complex and did not support some of the use cases (single row insert or update). Merge is available in other DBs (Oracle?)

Why you want Upsert:

  • works for large sets of rows
  • supports complex logic
  • supports cascading
  • rewriting procedural ETL logic as single SQL statements
  • faster (sometimes)

How it works -- It probes the index (which means BTree index), (caveat?), locks the leaf page. If we try concurrent upserts they will fail because of lock, but will not overwrite data (??).

BRIN -- Block-Range INdex:

  • new index type
  • also known as "skiplist" index
  • good for naturally ordered data.
  • WHY? - very small indexes for very large data. (BTree:2142MB::BRIN:(64kb for 512, 192kb for 64))
  • Table is divided into blocks. for each block store min and max value.
  • lot faster to load index. (second quandrant tested this using TPC-H Lineitem data; 95:BTree::18:BRIN)

GROUPING, SETS, ROLLUP and CUBE:

ROLLUP -- get subtotals and totals with detail in one query.

SELECT country, city, count(*)
FROM lwn_subscribers
GROUP by rollup (country, city);

CUBE -- explosiion of all summaries for use with OLAP tools (e.g: Mondrian, Pentaho etc.,). Earlier we had to do this externally (say Java). (CUBE output is not meant to be human-readable).

GROUPING SETS - superset/programmable version of both ROLLUP and CUBE.

SELECT ...
FROM ...
GROUP by grouping sets ((city, level), (level), ());

Abbreviated key sorting

  • sort varlena data by key prefix
  • much faster index build
  • 2x to 20x faster
  • works with TEXT, VARCHAR, ..
  • no extra work.. just upgrade to 9.5.

Foreign Schema

  • Foreign Data wrapper
  • let pg access external data like it was a local table.
  • example: redis (what are all my keys)
CREATE FOREIGN TABLE myredishash
(key text, val text[])
SERVER redis_server
...

INSERT INTO myredishash (key, val)
VALUES ('mytable:r1, '{prop1, val1..}');

FDW to query other PostgreSQL instances,Hadoop, cassandra, kafka, spreadsheet, twitter(REST), multicorn (Python).

Import Foreign Schema (SQL DBs):

IMPORT FOREIGN SCHEMA public
EXCEPT (reports, audit)
FROM server acctng_server into acctng;

currently on "postgresql-like" databases.

Foreign Table inheritence:

  • Data federation using postgresql paritioning
CREATE FOREIGN TABLE users_shard_1 ()
INHERITS (users) server shard_1;

CREATE FOREIGN TABLE users_shard_2 ()
INHERITS (users) server shard_2;

citusdb uses FTI to store in columnstore (cstore) tables.

More features:

SET LOGGED/UNLOGGED:

  • Use unlogged tables for ELT.
  • Google summer of code project
  • You can change logged/unlogged post-facto.
  • ALTER TABLE july_data SET LOGGED

GiST index-only Scan:

Index-only scan for PostGIS, Exclusion

WAL compression:

Reduce PostgreSQL log writes during data loading; less writing => faster load.

TABLESAMPLE:

Get a "quick look" at data in a big table.

SELECT * FROM user_profiles
TABLESAMPLE BERNOULLI (0.001);
  • Sampling algorithsm available now -- system, bernoulli.

Beyond 9.5:

Parallel seq scan; finally, parallel query for postgresql.

set max_parallel_degree = 4 ;
select * from pgbench_accounts
where filler like '%a%';

cstore & pg_shard

  • extensions from citusdata
  • cstore: column store for PostgreSQL
  • pg_shard: automated DW sharding -- automated sharding of tables to remote shards. Aimed at big data use case.
  • idea behind this: retrofitting to existing database instance that you never thought will get that big.

Big data forks go OSS

  • PipelineDB: August 2015. is a streaming database. "Spark, but relational"
  • Greenplum: November 2015. Still has the most mature "MPP", since it is the most battle-tested.
  • CitusDB: January 2016. Opensource as an extension. can be retrofitted to existing pg instl.

Questions:

  • "Recursive queries? (for graph databases)" - the perfect GraphDB is impossible. it can be implmented for one kind of graph. PostgreSQL does well on large forest of small bushes. Idea: "use PG for storage. Write a query planner for graph querying".

Links:

Additional reference:

notes

Hakyll

This website is was made using Hakyll and Pandoc. I have often tried to move away from Hakyll because I can't "hack it" (haha). But, very few other static compilers can leverage the power of Pandoc like Hakyll does. So, I'm going to use this opportunity to learn Haskell through Hakyll like many have.

Hakyll source code examples:


See also: haskell, pandoc

notes

Desktop

stuff related to desktop application development, distribution, µISV etc.,

  • Google Omaha Tutorial - fman Blog - "Google Omaha is the open-source version of Google Update, the technology which keeps desktop apps such as Chrome up to date." Uesful to know when you want to distribute a desktop application that needs to be kept automatically up to date.

  • Year in Review 2017 - fman Blog - a good example of a µISV.

web + desktop Development frameworks

Things in the "electron" space.

notes

Data Engineering

News

Software and Libraries

Articles

Books

Courses

Data Engineering Concentration - Data Science, MS | University of San Francisco

notes#data-engineering

CSCI 56500 - Programming Languages

Stuff related to CSCI 56500. I'm taking this course during Fall 2011 semester at IUPUI, taught by Prof. Rajeev Raje.

Text book: Advanced Programming Languages Design by Raphael Finkel. This book is available online.

Introduction

Recommended Reading

Control Structures

Semantics

Recommended Reading

Reference books

Related Courses

Prof Weimer of Virginia university is teaching CS 6610 - Design And Analysis Of Programming Languages in Fall 2011, which is quite interesting for a student of CSCI 56500.

Notes

Interesting links

  • It appears Lua gets the multiple return idea from Prof. Liskov's CLU language, which we studied in Chapter 2.

Open Questions

  • Do langauges that have S-expressions make it easy to implement continuations?

notes

CSCI 60300 - IUPUI CS

Papers read as part of "Advanced Distributed Systems" - CSCI 60300, Fall 2013 class taught by Dr. Raje at IUPUI.

  1. [Beugnard, Antoine, Jean-Marc Jézéquel, Noël Plouzeau, and Damien Watkins. "Making components contract aware," 1999. 32 (7)http://dx.doi.org/10.1109/2.774917.]{.citation cites="Beugnard1999"}; PDF.
  2. [Beugnard, Antoine, Jean-Marc Jezequel, and Noel Plouzeau. "Contract aware components, 10 years after," 2010. http://dblp.uni-trier.de/db/series/eptcs/eptcs37.html#abs-1010-2822.]{.citation cites="abs10102822"}; PDF
  3. [Tilak, Omkar J, and Rajeev R Raje. "Temporal interaction contracts for components in a distributed system," 2007..]{.citation cites="tilak2007temporal"}; PDF
  4. Synchronization Level Specification and Matching of Software Components by A. Kumari, K. Pradhan, L. Gallage and R. Raje
  5. Trust Contract of a Service and its role in Trustworthy Service Selection by L. Gallege, D. Gamage, J. Hill and R. Raje
  6. [Gamage, Dimuthu U, Lahiru S Gallege, James H Hill, and Rajeev R Raje. "A compositional trust model for predicting the trust value of software system qos properties," 2012..]{.citation cites="gamage2012compositional"}; PDF
  7. [Bracciali, Andrea, Antonio Brogi, and Carlos Canal. "A formal approach to component adaptation," 2005. 74 (1).]{.citation cites="bracciali2005formal"}; PDF
  8. Multilevel Specification for Adaptive Service MS Thesis by S. Phatak .
  9. [Lamport, Leslie. "Time, clocks, and the ordering of events in a distributed system," 1978. 21 (7).]{.citation cites="lamport1978time"}; PDF
  10. [Chandy, K Mani, and Leslie Lamport. "Distributed snapshots: Determining global states of distributed systems," 1985. 3 (1).]{.citation cites="chandy1985distributed"}; PDF
  11. [Dabrowski, Christopher, Kevin L Mills, and Stephen Quirolgico. A model-based analysis of first-generation service discovery systems, 2005..]{.citation cites="dabrowski2005model"}; PDF
  12. [Pradhan, Ketaki A, Lahiru S Gallege, and Rajeev R Raje. "MDE-urds-a mobile device enabled service discovery system," 2011..]{.citation cites="pradhan2011mde"}; PDF
  13. [Han, Taekgyeong, and Kwang Mong Sim. "An ontology-enhanced cloud service discovery system," 2010.. Vol. 1.]{.citation cites="han2010ontology"}; PDF
  14. [Chandy, K Mani. "Caltech infospheres project overview: Information infrastructures for task forces," 1996. 256.]{.citation cites="chandy1996caltech"}; PDF
  15. [Foster, Ian, Carl Kesselman, and Steven Tuecke. "The anatomy of the grid: Enabling scalable virtual organizations," 2001. 15 (3).]{.citation cites="foster2001anatomy"}; PDF
  16. [Foster, Ian, Carl Kesselman, Jeffrey M Nick, and Steven Tuecke. "The physiology of the grid," 2003..]{.citation cites="foster2003physiology"} PDF
  17. [Chandy, K Mani. "Sense and respond systems," 2005..]{.citation cites="chandy2005sense"}; PDF
  18. [Dean, Jeffrey, and Sanjay Ghemawat. "MapReduce: Simplified data processing on large clusters," 2008. 51 (1).]{.citation cites="dean2008mapreduce"}; PDF
  19. [Fox, Armando, Rean Griffith, A Joseph, R Katz, A Konwinski, G Lee, D Patterson, A Rabkin, and I Stoica. "Above the clouds: A berkeley view of cloud computing," 2009. 28.]{.citation cites="fox2009above"}; PDF
  20. [Weiser, Mark. "The computer for the 21st century," 1991. 265 (3).]{.citation cites="weiser1991computer"}; PDF

notes#distributed-systems#papers

Computer Architecture

Register allocation is a NP-hard problem. See "Register Allocation:What does the NP-completeness Proof of Chaitin et al. Really Prove?" via this thread on RTX2010 – Radiation-hardened stack machine microprocessor

What every programmer should know about memory, Part 1 [LWN.net] by Ulrich Drepper; September 2007. 9 part series.

Why is Apple’s M1 Chip So Fast?. Real world experience with the new M1… | by Erik Engheim | Nov, 2020 | Medium; Nov 2020.

Putting the "You" in CPU Curious exactly what happens when you run a program on your computer? Read this article to learn how multiprocessing works, what system calls really are, how computers manage memory with hardware interrupts, and how Linux loads executables.

notes#computing#hardware

Notes on Aphyr blog posts

These are my notes on reading Kyle's very informative series of blog posts on Jepsen to test various distributed databases. Errors are my own.

Jepsen

281. Jepsen: On the perils of network partitions

What is linearizability? From Linearizability versus Serializability,

"Linearizability is a guarantee about single operations on single objects. In plain English, under linearizability, writes should appear to be instantaneous. Imprecisely, once a write completes, all later reads (where “later” is defined by wall-clock start time) should return the value of that write or the value of a later write. Once a read returns a particular value, all later reads should return that value or the value of a later write."

Uses a cluster for 5 machines on a LXC based virtualization to simulate the network partitions using Jepsen.

salticid, a fabric(?) like remote execution tool is used to run the tests on the cluster.

How to create a partition? cause nodes n1 and n2 to drop traffic from n3, n4, and n5.

Other errors that are used to simulate network problems:

  1. partition
  2. slow
  3. flaky (probabilistic dropping of messages)

The test itself is simple. Use 5 clients each to generate a sequence of non-overlapping list of numbers. Write these numbers to the database. See how many of these were actually saved to the database. Get the difference as the "loss".

282. Jepsen: Postgres

PG accepts connections on a single primary node, which means, if you can't talk to that primary node, the system is unavailable. This means PG is Consistent and has Partition tolerance (CP).

"Even though the Postgres server is always consistent, the distributed system composed of the server and client together may not be consistent. It’s possible for the client and server to disagree about whether or not a transaction took place." The 2phase commit (2PC) protocol says that we must wait for the acknowledgement message to arrive in order to decide the outcome.. If the client does not get the ack from the primary node, the system has a deadlock.

Strategies for dealing with 2PC(on the db, in this case - PG), (when we suspect network partitioning..):

a. Accept false negatives -- say the write failed even if there is a small chance of success b. use data structures to allow idempotent operations. This way, when you encounter network errors, you can retry the write. A queue that has "atleast once delivery" semantics is a good place to put repeatable writes that can retried later. c. write the transaction ID to the DB or a log on a disk or atleast-once-queue. Once the network partition is resolved, retry or cancel (if the transaction had succeeded in the earlier attempt).

"A network partition–and indeed, most network errors–doesn’t mean a failure. It means the absence of information."

283. Jepsen: Redis

Redis as often used a message queue, lock service, session store, or even as database. Redis in a single-server setup is a CP system. Redis ofers asynchronous primar -> secondary replication, which means that the primary returns the response to the writing client even before the replication to the secondary is complete.

A group of nodes is also called a component.

  • ??How to write the verification of a consensus algorithm in TLA+??

Programming

Clojure

notes

Apache Phoenix

Notes

Where does Phoenix fit in?

phoenix demo

  • 10 years of Fortune 500 stocks
  • 30M rows
  • Single AWS node
  • interactive visual query for stock prices ...

What is Hbase

  • Runs on HDFS
  • K/V store -- distributed, sorted, map
  • low level APIs -- get, put values given row key & scan over range given start and stop row keys.

Why use HBase?

  • you have lots of data. scale linearly. shards automatically. data is split and redistributed
  • no transactions out of the box.
  • you need strict consistency.
  • hbase handle incremental data changes by building log structure merge trees

Why use Phoenix?

  • SQL

  • reduces the amount of code users need to write.

  • Performance optimisations transparent to user

    • aggregation
    • skip scanning
    • secondary indexing
    • query optimisation
  • Leverage existing tooling around SQL

    • SQL Client (bin/sqlline.py)
    • OLAP engine. eg: Pentaho. Kylin?

Phoenix versus Hive performance

  • Phoenix (key filter) >> phoenix (full table) > >> Hive over HBase
  • Hive and Impala have similar perf. Does not understand the structure of row-keys

Use cases

  • Data archival

    • from oracle into HBase while maintaining salesforce developer API
  • Platform monitoring

Why is it important?

  • scalable, low latency app development starts with Phoenix
  • Phoenix worries about physical scale and fast performance so that app developers don't have to
  • Looks, tastes, feels like SOQL to force.com developer

Archival problem set

  • Field history tracking grows unbounded. call centers call individual change. No data is deleted for 8.5 years.
  • Enterprise customers require long term storage of 'cold' data.
  • Data retention policies can require of data to be kept around

Archive pilot demonstration

  • insret data from Oracle into Phoenix

Monitoring use case

  • security, compliance and audit
  • product support and "Limits analysis"
  • product usage and management (roll up into aggregates)
  • Identity fraud

Phoenix under the hood

  • aggregate co-processor on severside per region, parallelised.
  • returns only the distinct set of features
  • preserves network bandwidth
  • client does final merge-sort.
  • client returns the result as a JDBC result

Roadmap (Nov, 2013)

  • Apache incubator
  • Joins
  • multi-tenant tables
  • monitoring and manageent
  • cost-based query optimizer
  • sql OLAP extensions (WINDOW, PARTITION OVER, RANK)
  • Transactions

State of the union (Jun 2015)

  • Runs TPC queries

  • Supports JOIN, sub-queries, derived tables

  • Secondary indexing strategies

    • Immutable for write-once/append only data (eg: Log data.. has time component). There is no incremental secondary index.
    • Global for read-heavy mutable data (co-processors on the server side keep indexes in sync with data tables). A second HBase table is created that is ordered according to the columns you are indexing. Optimiser will use this table if it deems this table to be more efficient.
    • Local for write-heavy mutable or immutable data. Custom load-balancers. All the writes are local. The writes can be done on the same region servers.
  • statistics driven parallel execution -- background process in hbase. allows the user to optimze queries. uses equidistant keys in region to collect stats.

  • Tracing and metrics for monitoring and management -- better visibility into what's going on. Uses Apache HTrace. Collect data across scans into stats table. Then query for bottlenecks -- index maint etc.,

Join and subquery support

  • Left/Right/Full outer Join; cross join

  • additional: semi join; anti-join

  • Algorithms: hash-joni; sort-merge-join

  • Optimizations:

    • Predicate push-down
    • FK-to-PK join optimisation
    • Global index with missing data columns
    • Correlated subquery rewrite
  • TPC - benchmarking suite

  • HBase 1.0 support

  • Functional indexes -- 4.3 release of Ph. Indexing on expression not on column.

SELECT AVG(response_time) FROM SERVER_METRICS
WHERe DAYOFMONTH(create_time)=1

Adding the following index will turn it into a range scan:

CREATE INDEX day_of_month_idx
ON SERVER_METRICS(DAYOFMONTH(create_time))
INCLUDE(response_time)
  • User defined functions -- extension points to phoenix. Define the metadata of the fun (input, output) params.
CREATE FUNCTION WOEID_DISTANCE(INTEGER, INTEGER)
RETURN INTEGER AS 'org.apache.geo.woeidDistance'
USING JAR '/lib/geo/geoloc.jar'

usage:

SELECT FROM woeid a JOIN woeid b on a.country = b.country
WHERE woeid_distance(a.ID, b.ID) < 5;

This query executes on server side. This is tenant aware, which means.. there is no chance of conflict between tenants defining the same UDF.

  • Query Server + Thin Driver -- offloads queryplanning and execution to different server(s). Minimises client dependencies

  • connection string: Connection conn = DriverManager.getConnection("jdbc:phoenix:thin:url=http://localhost:8765");

  • See http://phonenix.apache.org/server.html

  • Union ALL support -- query multiple tables to get single resultset.

  • testing at scale with Pherf -- define a scenario and deploy pherf to the cluster. It will generate the data and collect the stats. Testing with representattive data size. functional data sets.

  • MR Index build

  • Spark Integration .. RDD backed by a Phoe table.

  • Data built-in functions -- WEEK, DAYOFMONTH etc.,

  • Tranactions -- using http://tephra.io

    • supports repeable_read isolation level
    • allows reading your own uncommitted data
    • Optional -- enable on a table by table basis
    • no performance penalty when not used
    • Optimistic concurrency control
    • no deadlocks or lock escalation
    • cost of conflict detection and possible rollback is higher
    • good id conflicts are rare: short transactions, disjoin partitioning of work
    • conflict detection is not necessary if the data is immutable. ie., write-once/append-only data.
  • Tephra Architecture

Apache Calcite

  • next step for Phoenix
  • Query parser, compiler and planner framework
  • SQL-92 compliant
  • Pluggable cost-based optimizer framework
  • Interop with Calcite adaptors
  • already used by Drill, Hive, Kylin, Samza
  • Supports any JDBC source
  • One Cost model to rule them all

Installation

Setting up a local phoenix cluster using Docker

Download and install Docker beta for Mac

We will setup a three node hadoop cluster using this recipe as a template.

git clone git@github.com:kiwenlau/hadoop-cluster-docker.git
cd hadoop-cluster-docker
docker build .

Lot of things fly by ...

Successfully built ed4ece2b19f2

Create hadoop network

sudo docker network create --driver=bridge hadoop
Password:
f678bdfc5918e15a578b909692e6f04a4ef5f730a95ebb0f16da5a30c38354b1

notes

AI Agent Framework

AI Agent frameworks:


Infrastructure

Ensue.dev --  is the shared memory and coordination harness for AI agents. It provides a persistent, semantic layer where agents can store observations, share context, and coordinate actions across any tool, model, or framework. Instead of isolated, stateless agents that start from zero each time, Ensue enables agents to build on each other's work. One agent's insight becomes another agent's starting point. See also: Ensue network / autoresearch

notes

1998

A Vedanta Toolkit

Sources: Article: A Vedanta Toolkit Part 1 and Article: A vedanta toolkit part 2

Author: Subhanu Saxena; profile

At long last, things have quietened down in Russia for me to post the first of 3 articles I had discussed in my new member introduction, namely "A Vedanta Toolkit". The subsequent articles "Shankara's AdhyAsa bhAshyam-a guided tour" and "A Shankara reader-key passages from Shankara's bhAshya's that encapsulate his teaching" will follow over the next few months.

A Vedanta Toolkit: Introduction

This genus of this article is a series of talks I had given in London to students who were about to embark on a study of Shankara's BhAshyas, GauDApAda's kArikas, and Sureshwara's writings. Many found it extremely helpful to have understood the following basic concepts before delving into the texts themselves. I shall try and be as succint as possible, as my intent is to raise awareness/curiosity in readers so they feel compelled to study the texts for themselves to confirm/deny what comes below. It is evident from recent postings that there are a number of members already familiar with these concepts. To them I look forward to their own insights. To those approaching the concepts for the first time, I hope the below proves useful.

I will cover the 4 concepts of "sarvArtrika anubhava", "adhyAsa", "vyavahAra and paramArtha drishti" and "adhyAropa-apavAda prakriyA". A number of others also add "vastutantram jnAnam" and "sAkhsi anubhava". I will treat the second as part of the discussion on anubhava, and leave vastutantram jnAnam as a natural by-product of a correct understanding of adhyAsa.

  1. SarvAtrika anubhava

Translated as universal experience or intuition, this sets vedanta apart from other systems. Vedanta is not a speculative system relying upon pure reasoning or faith, but is based on a universal experience that has been and always will be available to us all. Using modern scientific terminology, Vedanta affirms a repeatable verifyable experiment rooted in actual experience vs belief in something outside of us. This is why the Upanishads do not attempt to prove/disprove the existence of Atman, as it is our very self that cannot be denied. All that is required is to remove our misconceptions, so Atman can rest in Atman free from distinctions (see later). Hence vedanta does not teach it as "something to be attained", or some magical transformation of reality. It is ever present as an experience.

Shankara also appeals to this unversal experience as he guides us to the truth. Examples of this are:

  • We all have the experience of mistaking one thing for another
  • We all have the experience of a state where distinctions of knower, known, means of knowledge do not exist, ie deep sleep
  • We all have the experience of understanding the true nature of something when our misconceptions are removed.

Post Shankara commentators have often ignored this basic tenet in interpreting Shankara, and indulge in dialectics that stretch Shankara's true interpretation.

So, concept number 1 allows us to stay on the straight and narrow and allows us to tolerate other beliefs and points of view, because the ultimate arbiter of who is right comes from the experience itself, not the word.

  1. AdhyAsa

If you want a thorough understanding of the message of the Upanishads, and the purpose of the shAstra, then you need a true understanding of adhyAsa. The text "par excellence" is Shankara's introduction to the Brahma Sutra BhAshyam, often called the adhyAsa bhAshyam. Since it will be the topic of a separate article, all I will say now is that it never ceases to amaze me how little known this astonishing text is amongst non-vedantins and the general public. It a short encompass, Shankara lays the ground work for all we need to know when approaching vedAnta. Having committed the text to memory, I often found myself reciting it (much to the annoyance of my children: "oh no, there goes Daddy with that Shankara thing again") as it helps keep me focussed on the key issues. Anyway, enough of a digression. More in my second article.

Shankara crisply summarises the concept at the beginning of adhyAsa bhAshyam as "MithyAjnAnanimittaha satyAnrte mithunee kritya, ahamedam, mamedam iti naisargiko'yam lokavyavahAraha" "Based on a false conception, the mixing of the real with the not real such as 'I am this' and 'this is mine' is a natural innate phenomenon in the world". Simply put adhYasa, meaning superimposition, is confusing the self with the not self, and mistaking something for something else. The result of this superimposition has many manifestations,

  • The distinction of subject and object, and the mixing up of the 2
  • The distinctions of a knower (PramAtr), the known (Prameya), and the means of knowledge (PramAna). For those interested, I can post the sanskrit definitions of these from nyAya darshana in a separate posting if they so request.
  • The presupposition that one is an agent/a seeker/a doer etc distinct from the Eternal Witness or sAkshi.

This adhyAsa is the precise definition that Shankara gives to avidya. Wherever there is this distinction of pramAta, prameya, pramAna, or Veditr, vedya, vedanA there is avidya. The correct ascertation of things as they are (vastutantram jnAnam as opposed to kartrtantram jnAnam, or knowledge dependent on the subject) is called vidyA. The duality of name and form thus fashioned by avidya is defined as mAyA.

It is important here to be clear what Shankara does NOT say avidya is, but has been presumed to say by later commentators:

  • Shankara does not imply the existence of an unreal undefinable avidyA shakti or mulAvidya as followers of the panchapAdika and vivarana commentaries claim. The source of the confusion is a difference in interpretation of the word "mithyAjnAna" in the above phrase. Shankara means "mithyA" + "jnAna", or false knowledge in the sense of misconception. In the panchapAdika this word is broken as "mithyA"+"ajnAna", or unreal ignorance. This is postulated by followers of this school as the material cause of the universe that clings to everything and is this mysterious "mAyA" that becomes the catch-all for everything we cannot explain. This interpretation is in direct contrast to repeated statements by Shankara in many other places in his Bhashyas that make it crystal clear that that is not what he meant. As but one example, Shankara frequently uses the word "mithyApratyaya" as a synonym for "mithyAjnAna". It is impossible under the rules of Sanksrit to decompose this word into anything other than "MithyA"+"pratyaya", or a false conception. Shankara is therefore not referring to some inscrutable power that is veiling the world. For him that would be contrary to concept 1, that of sarvArtrika anubhava.

  • Shankara never equates avidya and mAyA, unlike later commentators. I will give the full quote in my 3rd article, but the reference here is to an unambiguous statement at Brahma Sutra BhAshya 2-1-14 "sarvajnasya Ishwarasya ...."

Correct understanding of the above makes it clear why Shankara and Sureshwara never trouble themselves with questions pertaing to the cause of avidyA, or to whom does avidyA belong. For anyone under the clutches of avidyA does not know that they are , and once knowledge has arisen, there is no more avidyA! Shankara has a curt response to the question "to whom is avidyA" at B S Bh 4-1-3, where heaffirms the above. Avidya belongs to whom it appears to belong.

A correct understanding of Shankara's position on Avidya makes it all the more obvious why he regards all vidhi's, secular activities including Vedic, and treatises on liberation as in the clutches of avidyA, as they all uniformly presuppose the concept of an agent, a doer, or a seeker which, as we have seen above, is avidyA. The full quote on this comes from B S Bh Introduction: "tametam avidyAkhyam AtmanAtmanoho itareta adhyAsam puraskrtya, sarve praMana-premeya vyavhahArAha laukikA vaidikAscha pravrttAha. sarvAni cha shAstrAni vidhipratishedha-mokshyaparAni". As such Shankara sees the function of shastra as simply removing avidya in the form of misconceptions of the notions of agency and knowership rather than pointing out some external reality. Hence he calls the shAstra antyam pramAnam, or final pramAna. Once knowledge dawns, what possible use could there be for further pramAna's.

  1. VyavahAra and paramArtha drshti

Having laid the groundwork of AdhyAsa, we can turn our attention to the two standpoints adopted in the shAstra, and the writings of Shankara, GaudapAda and Sureshwara: The arguments found in these sources of literature can be dividied into a standpoint where the existence of a distinct knower and something to be known by some means of knoweldge exists, and where such a distinction does not exist. Shankara uses the terms "(loka)vyavahAra" and "paramArtha" respectively. These are commonly translated as the empirical and transcendental standpoints respectively. I often hesitate at using the word "transcendental" as it has a certain flaky imagery in modern English. I tend to prefer describing them as the empirical, where there is apparent reality of the existence of cause and effect, and the "really real" where such distinctions do not exist.

A number a discussions/commentaries on the Upanishads and Shankara fail to recognise from which standpoint a discussion is being advanced, and they will frequently try and justify a viewpoint expressed from an empirical standpoint as pertaining to some ultimate truth. As an example, we have questions such as "is there conscioussness in deep sleep?", or "is there any avidyA in deep sleep?" when the the avasthA traya discussion is simply a concession to the empirical standpoint, whose teaching is ultimately rescinded (see concept 4). An interesting corollary of the above is that Shankara would have no problems with our modern understanding of the laws of Physics. For him the empirical standpoint is a standpoint where the laws of cause and effect must operate, so there must be a set of physical laws that govern there. Equally, from the transcendental standpoint, if there really is a unified field, then the effect can be no difference than the cause, as there is no real duality of physical laws, only apparent duality.

I refer readers to an extremely helpful table in"Shankara's clarification of Vedantic Concepts" by Holenarsipur Swami Sri Sri Swami Satchidanandendra Saraswati that compares and contrasts the concepts of Atman, avidyA, moskhA etc from both the empirical and transcendental viewpoints.

  1. adhyAropa-apavAda prakriyA

The previous 3 concepts allow us to really understand the underlying method in all the Upanishads, namely that of adhyAropa-apavAda, or the method of deliberate superimpostion and recission. It helps us get past the questions of "how do objectify the unobjectifiable" and "how can you describe the indescribable".

The method is directly quoted by Shankara in his Bhagavad Gita Bhashya 13-13/14 referring to the Slokas "sarvataha pANipAdam tat...." and "sarvendriya guNAbhAsam....". Here Krishna describes the absolute as both having arms and feet everywhere, and also as having no distiguishing features whatsoever. Lest we scratch our heads indefinitely over this, Shankara calms us down by stating that this is nothing other than the application of the basic method adopted in the shAstra, and he says, quoting a half verse from the ancient tradition: "tathA hi sampradAyavidAm vachanam 'adhyAropa-apavAdAbhyAm, nishprapancham prapanchyate' ", translated into English as "so say the followers of the tradition 'that which is beyond all manifestations is made manifest by the method of deliberate superimposition and recission' ". The term adhyAropa apavAda is none-other than the technical term given to the method "neti neti" or "not this not this" adopted in the Upanishads. GaudapAda refers to the method in his own words in the kArikA's:

Sa ya esha neti netIti vyakhyAtam nihnute yataha sarvam agrAhyabhAvena hetunAjam prakshyate.

"Through this method of neti neti, where what has been said is subsequently removed, the unborn beyond conception shines forth of its own"

In the BrihadArnyaka bhAshya vArtika, it is clear that sureshwAra was also intimately familiar with the method. Witness BBV 2-3-171, 199, and many others.

Briefly, the method deliberately superimposes some characteristic (such as "satyam, jnAnam anantam brahma" in Taittiriya Up), to remove some misconception, only to deliberately rescind the teaching when it has served its purpose (eg "ashabdam asparsham.."). For example the quote from Taittiriya above is adyAropa to remove the misconception that Atman is limited in scope. The teaching is rescinded at the end of the pancha kosha discussion in that Upanisad, when all that is left when distinctions are removed is Ananda, or pure bliss. Similarly, the concept of an individual self is superimposed to remove the misconception that the body is the real self. This teaching is ultimately rescinded, lest the seeker actually think he is an individual soul undergoing rebirth, as in the kArikA "marane sambhave chaive, gatyAgamanayor api. Sthhitau sarva shareereshu AkAshenAvilakshaNaha".

Those familiar with modern science will also be aware of the method. When students learn about the structure of the atom, they start with a description of a "planet(nucleus)" and a "moon(electron". Later this teaching is replaced by a teaching of a wave around the nucleus, followed by a wave particle, in defined energy states etc etc. Each stage of the teaching replaces a previous superimposition with a more refined view. This is not dissimilar to the method of adhyAropa-apAda, which essentially vectors the seeker in the direction of the truth. Once all misconceptions have been removed, there is no need for a final "there it is", as the truth will manifest itself of its own accord.

Shankara gives an example in his Brihad BhAshya from ancient times, whereby priests show to married couples the Arundhati star in the night sky. The approach he describes is that first the priest points to the sky in the region of Arundhati and designates a bright star as Arundhati. He then will say "no, that's not it" and point to another less bright star, and so on, until he finally reaches Arundhati.

Examples of the method are many in the Upanishads:

  • The doctrine of panchakosha, or 5 sheaths. The teaching is not that there is literally layer upon layer shrouding the inner Atman but, as a device to lead the seeker, koshas are superimposed and then rescinded until what is left is Ananda, the anubhava or adviteeya Atman. -Examination of the 3 states of conscioussness. The teaching appears to analyse each state of conscioussness as some manifestation of reality, leading a seeker to the state of non-duality where no distinctions are left. The final teaching of "Turiya" is not an independent 4th state, but a rescission of the previous 3 states as having only been superimposed on the Atman temporarily for the teaching, and a device that finally is discarded. Failure to appreciate this in the method is probably the key cause of misinterpretations by later commentators.

  • Teaching on creation. Again the superimposition is that Brahman is the cause of the universe, which is done to remove the misconception of the independent existence of the universe. This teaching is then rescinded by stating from the paramArtha standpoint the effect is non different from the cause, and all there is is Brahman. So, we have the kAraka "mrlloha visphulingAdyaihi, strshtir yA choditAnyathA. upAyaha so'avatArAya nAsti bhedaha kathanchana" "By illustrations like those of clay, gold and sparks etc, creation is variously described by the Sruti. This is just a device to take the mind to the truth. There is no distinction (in Brahman) whatsoever."

  • Distinction of teacher and taught. Again, this is a device for instruction, for on the dawn of knowledge this distinction is seen to be only apparent. There is the famous kArakA: "vilkapo vinivarteta kalpito yadi kena chit. upadeshAdyam vAdaha, gyate dwaitam na vidyate" "should anyone fashion the construct (of teacher and taught) then it would have to be removed. This is only for the purpose of teaching, when knowledge dawns, there is no duality"

-Avidya itself. Again, it should not be assumed that the Sruti actually teaches there is some manifoldness that has to be removed. For, from the paramArtha standpoint, there is only Brahman. Again, GauDApA says "prapancho yadi vidyeta, nivarteta na samshayaha. mAyAmAtram idam dwaitam adwaitam paramArthataha" "If there really was a manifoldness in reality, then without a doubt it would have to be removed . But this duality is a false appearance, non-duality is the supreme truth".

Again, many post-Shankara commentators have taken a discussion that is really adhyAropa, and tried to fashion a more fundamental truth out of it, when the ultimate instruction is to rescind the teaching in hand. I am sure that other members will have their own personal favorite examples out there (eg the distinction of higher and lower Brahman etc etc), but the examples above should give people a feel what to look for as they read, and give them some perspective.

Conclusion

The above should give anybody wishing to delve deeper into Shankara's bhAshyas, gauDApAda or Sureshwara's vARtikA's and his naishkarmya siddhi a good grounding in the basic concepts around which the teachings are given. When I read a passage in the Upanishads, Gita, Brahma Sutra or the Bhashyas etc, I tend to automatically go through the following mental checklist to try and understand the message:

  • Is the empirical or transcendental viewpoint being adopted?
  • Is this adhyAropa, or apavAda of a previous argument?
  • If it is something written by a post Shankara commentator, have they captured the essence of Shankara's teaching on avidyA?
  • How does the line of reasoning accord with universal experience?

Well. that's all for now. Hopefully the above will prove useful to those who have not come across the concepts before. For those who have, it would be interesting to hear your insights/elaborations. A true understanding of the 4 concepts should help guide you through such questions as:

  • What is the purpose of shAstra?
  • Is punarjanma real, or only a device for instruction?
  • What does Shankara really think of injunctions, meditation etc?
  • Is moksha something to be attained by me? Is it now or after a die?

I leave you with the thought that I have written too much. As rshi Badhwa (quoted by Shankara) explains to BAshkali when he asks the question 2-3 times "what is Atman?" and Badhwa gives no response, he finally says in accordance with principle 1 (sarvArtrika anubhava) "you ask but you do not understand the instruction-this Atman is utter silence"

Harihi Om

notes