Notes on Software Development

by moodyharsh
software

Programming is an activity of designing Software Applications and Interactive Media.

A Software Application is a Control System with Digital Interfaces suited for the particular Domain.
Data, Components and Flows constitute the Constrol System of the Software .

Basically what happens with research is that it often omits numbers
in favour of elegance of formulas. In engineering you show numbers
in favour of a formula.

Modularity which is mainly about the reconfigurability of the system, as in how you can take it apart and put it back.

Assembly

OS
System Calls

Memory I/O
Grpahic Card I/O
Device Peripheral I/O
Network I/O

Concurrency
Arithmetic
Control Flow
Functions
Linkers

  • rendering, music processing, dsp, graphics pipe line, data processing in spark, network packets, request processing …

Programming Language Concepts

Namespaces
Templates
Structs
Pointers
ADTs
Serialisation
Immutablility
Lazy
Lexical Scope / Global Scope / File Scope
Curyying
Transpiler / Preprocessor / Compiler
Symbols
Monads
Continuations
List Comprehensions
Destructuring
Pattern Matching
Polymorphism
First Class Functions
Decorators
Literals - Numeric, String, JSON, Array, Dictionary, …
Loops
Infix Expressions
Statements
DSL
Macros
Type Declarations
Duck Typing
Modules
Blocks
Concurrency
Coroutines
Comments / Doctstrings
Exceptions
Multiple / Single Dispatch
Invariants / Assertions / Data Guards
Operator Overloading
Annoatations
Early bound - actions / inheritance -> Type Checking -> Design Patterns
Late bound - messages / delegation -> Duck Typing -> Reflection and Closure
OSV Syntax

Monads

State can be mimiced in haskell by passing around the changed value.
Order of execution can also be mimiced by passing function as an argument.
All monadic functions interract with compiler in one way or the other, like perl’s taint.

Platforms

OS/Mobile/Watch/Smart TV/Smart Device/Embedded/Browser/VR/AR

  1. Qt / Cocoa
  2. Java
  3. Unix
  4. HTML5
  5. Facebook / Twitter Platforms
  6. Github
  7. DOTA/Minecraft

Frameworks give you

  1. callbacks for events
  2. actions for getting and posting data
    with which you write extensions

Tie In with platoforms vs Independance

Platform -> App
Framework -> Extension

What is illegal on platforms

  1. Sex/Drugs/Hate
  2. Piracy
  3. Hacking/Spying/Malware
  4. Competing Platforms

An App or Extention is a combination of APIs given by the Platform/Framework which solves a use case

Databases

  1. Records | SQL, Embedded SQL, Distributed SQL
  2. Data structures | Redis
  3. KV, Distributed KV | Memcache, BDB, Others
  4. CSV | Hadoop
  5. Dataframes - Spark, Pandas, Druid
  6. JSON | Mongodb, CouchDb, Cassandra
  7. Files | File System

Scaling

Vertical

Horizontal

  1. Sharding
  2. Master / Slave

School

Mathematics
- Probability and Statistics
- Discrete Mathematics
- Operations Research
- Numerical Analysis
- Control systems

Algorithms and Data Structures
Theory of Computation
Compilers and Languages
OS
Computer Networks

C / C++ / Java / Python
Assembly Programming
OOP, Design Patterns
RDBMS
Parallel Computing
Storage Networks

Data

  • ER / Binary / XML / Object / EAV / File
  • Queries / Commands
  • Serialization
  • Storage and Caching
  • Indexing / Searching / Tagging / Learning
  • Compression
  • Caching
  • Ontology
  • Analytics
  • Events / Messages
  • Logs
  • ETL
  • Snapshot + Restore
  • State / Context / Session
  • Protocol / Format
  • Language + Parsing + Validation
  • Transactions

Patterns

Active Record
DTO
DAO
Lazy Loading
Stream
Data Oriented
Data Binding
Patter Matching
Maybe
Memento
Memory Pool
Visitor
Fluent
Iterator
Composite
Prototype

Concurrency

Double Checked Locking
Scheduling
Lock / Monitor
CAP
ACID
BASE
Active Object
Promise
Balking

Types: Document, Graph, Table, Dictionary, List, Array, Matrix

Master Slave
Replication and Shrading

Data Science: Crawling, Analytics, Statistics, Transformation, Predictions, Visual Modelling, …
Simulations

Components

An Active Component has a

  1. A life cycle
  2. A way to get inputs and interrupts
  3. An execution body
  4. A way to delegate outputs
  5. A way to trigger events and alerts

Execution body is made of
- System Calls and Instructions
- Decisions and Loops
- Arithmetic
- Passive Components

A Passive Component is just made of execution body.

A set of Related Components is a Module.
Module manages dependencies and Component locations as well.

Active Components can be,
Reactive
Asynchronous
Synchronous
Controller
Proxy
Feedback
Feedforward
Scheduler
Router
Filters
Validators
Transformers
Authentication
Authorization
Source
Sink
Worker / Producer / Consumer
Co-operative Tasks

User Interfaces

  • Forms / Widgets
  • Layouts
  • Scene
  • Graph

Flows

Flows are Data Driven on Demand Driven.
Flow occurs between component couplings.
Data flows across a coupling.

How to build components and flows ?

Function is a Passive Component.

Activte components can be built with the following,

  1. Recursion
  2. Mutual Recursion
  3. Co-Routines
  4. Automata
  5. Tables
  6. Threads / Processes / Active Objects
  7. Objects with a single public method

Passive Component Coupling
- Function Calls
- Function Pointers
- Continuations
- Promises
- Closures

Types of Procedures

Commands
Queries
Routines
Transformers
Extractors
Loaders
Loggers
Assertions / Invariants
Exceptions
Integration

Misc

Service Locator
Fluent
DSL

Design Patterns

Iterator
Visitor
Template Method
Statergy
State
Command
Observer
Chain of Responsibility
Fluent Interface

Interceptor based

Architecture

Interceptor, Pipes and Filters
Blackboard / Tupplespace
Message Passing
Flow Based
ECS
MVC, PAC, MVP
SOA, EDA, MicroServices
Signals / Slots

Active Component Coupling
- Messaging
- Memory
- Wire
- Interceptors / Delegates

User Interface Coupling
- Observers

Remote
- RPC
- Half Objects
- Request / Response

State

Never store state inside a Component. State should either be stored in Data or in transient Event objects.
Object Oriented Programming encourages storing State. DO NOT FOLLOW IT.

Characteristics of Software

  • Size
  • Speed
  • Bandwidth
  • Memory
  • Scale
  • Complexity
  • Leaks
  • Entropy
  • Cascades
  • Uptime
  • Cost
  • Training

Deployment

  • CI
  • Packaging
  • Build
  • Version Control
  • Horizontal and Vertical Scaling
  • Upgrading
  • Backup / Restore
  • System Administration and setup
  • Performance Tuning
  • Logging and Auditing
  • Long term Data Storage
  • Monitoring
  • Network Administration and setup
  • Maintain a Changelog
  • Tuning

Testing Software

  • End User Simulation
  • Stress Testing
  • Edge Case Testing
  • Security Testing
  • Component Testing
  • Component Internals Testing
  • Tracing
  • Logging
  • Documentation

Naming Conventions

Data / Events Should follow structural Naming Conventions.
Components are AdjectiveVerbs
Couplings are VerbNouns

Files should be organized by
- Base
- Data Models
- Components
- Passive Components
- Main Component
- User Interfaces
- Simulations

Analysis Skills

  1. Story Gathering / User Research / Requirements Gathering
  2. Prototyping
  3. Estimating
  4. Architecting / System Modelling
  5. Designing
  6. Data Modelling
  7. Composing / Decomposing
  8. Process Modelling
  9. AB Testing
  10. Diagrams - ER / DFD / SC
  11. Mockups
  12. Set Milestones and Cost
  13. Delegating
  14. Delivery Planning

Misc Skills

  1. Scripting
  2. Presenting
  3. Negotiating
  4. Writing
  5. Reverse Engineering
  6. Debugging

Technique is not a substitute for discipline and focus.

Feature Driven Development

Build feature list / requirements

Object, Action, Result
User stories
Mockups
BDD Specs

Design Data

Set Milestones and Cost

Plan and delegate to feature teams

Data Flow Diagrams
Actor Models
Couple User Interfaces to Observers and Observers to Components
Couple Components

Build by feature

Bottom up programming is a lost Art form. I’ll try to list down the principles of it.

  1. Build Components one at a time, independently of all other components
  2. Use the Component using a Component Shell / REPL
  3. Test the Component using Automation.
    3.1 Make Inputs / Outputs / Network / Sources / Sinks abstract
    3.2 Use a Module Loader to manage Component Location
  4. Model User Interface as Events so that they can be Simulated.

The advantages of a Component Shell / REPL are multifold

  1. In makes Develop / Interact cycle easy
  2. It makes Application Scripting possible from the Start
  3. It makes Testing stupid.

Anti Patterns

Golden hammer
Over engineering

Inter Platform
Square wheel
Gold plating
Magic Button
Buzz words
Shotgun
Big Ball of Mud
Buzz Words

Beyond P
Reinventing the Square Wheel
Inner Platform

Frontend

Design has been fairly commodotized.

Free

Material CSS
Metro UI
Bootstrap
Semantic UI

Paid

Theme Forest

Icons
Stock Photographs

Can all be bought or downloaded for free.

Interactive Design is new is a new breed of HTML5 design that focus on
Game like interactions.

Skills

Layout
Colors
Typography
Accessibility
Gimp
CSS / Sass
Accessibility
Sound Mixing
Readability

Data Visualisation

WebGL
D3

Books

Desig of Everyday things
Don’t make me think
Elements of Typographic Style
Designing Interactions

Responsive Design

  1. Set Aspect Ratio
  2. Get Window Size
  3. Draw Container
  4. Set Background
  5. Use Zoom

http://www.hcibib.org/
https://www.interaction-design.org/

ppi and dpi

see graphics are just pixels. how they appear on the screen is dependent on the ppi of the printing medium.
1/ppi is basically the width of the pixel.

thus the quality of the printer or the quality of the monitor can be be determined by how many pixels it can draw and thus by how small 1/ppi is.

the A* paper sizes are designed in the following manner
height : width = root2 : 1

A4 size is 8 1/4 ? 11 3/4 in inches

before going to printing fonts, we have to define point. 72 points make an inch. so if
so if you want to print

  • Hue == Color
  • Saturation, Muteness
  • Lightness Darkness (Accent)
  • Tint (white), Shade(black), Warm(Red), Cool(Blue)
  • Neutral Colors - White, Black, Grey

The main idea of colors is to create contrasts. Contrasts are used to direct the users attention.

Typography

https://www.modularscale.com/
https://basehold.it/
http://lorempixel.com/

Brute Force

/* Large desktop */
@media (min-width: 1200px) { … }

/* Portrait tablet to landscape and desktop */
@media (min-width: 768px) and (max-width: 979px) { … }

/* Landscape phone to portrait tablet */
@media (max-width: 767px) { … }

/* Landscape phones and down */
@media (max-width: 480px) { … }

Concepts

grid layout
fixed
fluid
elastic
responsive
golden ratio

color palette

baseline
typography
colors
spacing

css tricks
shadows
rounded corners
sprite
minify
overlay
opacity
transitions
hover styles

aesthetics

Frameworks

960
gloden grid
blueprint
skeleton
yaml
foundation
cascade
http://grid.mindplay.dk
bemwhats it to you pork face ?
* functions – smallest divisible unit, avoid repitition
* vertical rhythm
* explain to 6 year old

Debugging

https://en.wikipedia.org/wiki/Five_Ws
https://en.wikipedia.org/wiki/5_Whys

  1. Finding errors as quickly as possible
  2. Keep errors from cascading
  3. Keep errors visible
  4. Add redundancy to avoid single point of failures
  5. Painless replacements

Time Analysis

  • Linear O(n)
  • Constant O(1)

Logarithm times

O(log n) - Binary Search

2, 2^2, 2^ 3 …..
0, 1, 2, 3 …

log (8,2) = number of steps taken on the number line to count the geometric progression.
See vihart’s video.

O(n logn) - Quick Sort

Each step has n comparisons.
Total steps = log n.

Data Structure Optimisations

  • Cache levels
  • Loop Unrolling
  • Branch prediction
  • AoS vs SoA
  • Slab allocator

Trees

  • parent and child
  • root and leaf
  • node and edge
  • sibling
  • ancestor and descendants
  • level
  • path
  • height == depth

Degree of node we call the number of direct children of the given node.
Branching factor is the maximum of the degrees of all nodes in the tree.

Tree is a recursive structure.

Depth of node is depth of parent plus 1.
Height of internal node is maximum height of height of internal node is maximum height of children + 1

Graphs

  • Nodes or Vertices
  • Edges (can be directed or undirected)
  • Weighted Edges

number of edges in a directed graph can be a maximum of n(n-1).

dense and sparse graphs.

path: each adjacent pair is connected by an edge. (trail)
simple path: vertices are not repeated (walk)
path.

connected graph: a path exists between any vertex.
strongly connected, weakly connected for diagraphs. weak means diagraph as graph is connected.

cycle: first and last nodes are repeated. tree is acyclic.

simple graph if there are no cycles.

representation: edge list
big o for operations - O(n2) where n is number of vertices

represenation: v x v adjacency matrix
representation: list of list of adjacency

proportional to O(v)

simple problems

  1. adjacent
  2. does a path exist

does a path exist can be found by DFS and BFS.
DFS is recursive. recursively goes deeper into the children.

BFS uses a queue and iteratively visits the adjancent nodes.

connected: there is a path b/w/ any two vertices
articulation point - deleting a node means the graph is not two graphs.
biconnected - no articulation points

kosaraju’s algo

G / G transpose

DFS on G transpose. Then sort by denominator.
DFS on G using vertices from previous. Every time you get a strongly connected component set.

tarjans algorithm

prims algo : spanning tree
Kruskal: minimum spanning tree

dijkstra: shortest path

In a game like AoE, divide the map into regions.
Run pathfinding algorithm to see if a path exists.

Dividing a graph into regions.
pick node.
tranverse all edges, while ignoring seen nodes.
if any nodes are remaining, create a group, repeat.

Depth first transverse each node edge first.
Use a stack to keep track.

Breadth first uses a queue but the stack no longer holds path.

Dikstra’s algo works for weighted graphs.
It is breadth first with minimum cost calculation.