Andrej Karpathy (Tesla)

Andrej is currently Senior Director of AI at Tesla,
 and was formerly a Research Scientist at OpenAI. His educational materials about deep learning remain among the most popular.
Programming The Software 2.0 Stack

Andrej's Landmark Post - "Software 2.0"

  • Software 1.0 consists of explicit instructions to the computer written by a programmer.

  • Software 2.0 can be written in much more abstract, human unfriendly language, such as the weights of a neural network.

  • In software 2.0, we restrict the search to a continuous subset of the program space where the search process can be made efficient with back-propagation and stochastic gradient descent.

Source: https://medium.com/@karpathy/software-2-0-a64152b37c35

Programming In The 2.0 Stack

  • If optimization is doing most of the coding, what are the humans doing?

    • 2.0 programmers label data

    • 1.0 programmers maintain the surrounding "dataset infrastructure":

      • Visualize data

      • Create and edit labels

      • Bubble up likely mislabeled examples

      • Suggest data to label

      • Flag labeler disagreements

  • Data labeling is highly iterative and non-trivial.

    • Lane lines are different across the world.

    • Cars have different shapes and sizes.

    • Even traffic lights and traffic signs can be ambiguous.

  • Label imbalances are very frequent.

  • Data imbalances are very common.

Realistic datasets: high label and data imbalances, noisy labels, highly multi-task, semi-supervised, active.

2.0 IDEs

  • Show a full inventory and statistics of the current dataset.

  • Create and edit annotation layers for any data point.

  • Flag, escalate, and resolve discrepancies in multiple labels.

  • Flag and escalate data points that are likely to be mislabeled.

  • Display predictions on an arbitrary set of test data points.

  • Auto-suggest data points that should be labeled.

Can we build GitHub for Software 2.0?