A tour through Yip!

This is a living document, as it is has not been finished yet. Tread carefully, here be dragons.

Reading Yip!

Yip! source is a flat sequence of tokens.

Yip!'s comments use octothorpes. An octothorpe followed by matching parentheses, brackets, or braces is a form comment, used to comment out entire composite literals in Yip!, and an octothorpe followed by any other character is a single-line comment:

# Single-line comment, runs to the end
#( form comment )

Form comments can be nested, and don't count opener/closer characters inside string literals:

#( a b #[ c "[d e]" f ] g h )

The comment convention #=> etc... is used to show that after executing its line, it will have output to the console.

Yip!'s annotations use backslashes. A backslash followed by matching parentheses is an annotation.

Throughout Yip! code, the form ( a -- b ) in comments and annotations describes stack effects. You should read this as "take a, and leave b". A stack effect of ( a b c -- d e ) is read as "take a, b and c, and leave d and e".

: word \( a -- b ) ... ;

Yip!'s syntax is sigil-driven, so it helps to know what every sigil means:

[...]         quotation (code as a value)
{...}         object literal
(...)         array

:foo          symbol
.foo          send message "foo"

^foo          lift "foo"

>foo          stash to stack "foo"
foo>          unstash from stack "foo"
foo.          peek from stack "foo"
foo/          drop the top of the stack "foo"
foo~          return whether the stack "foo" exists and has values

: ... ;       define a word
:= ... ;      evaluate body immediately and bind top of the stack

word: ... ;
word: ... !   command syntax sugar

.word: ... ;
.word: ... !  message command syntax sugar

.[...]        focus syntax sugar

module/word   namespacing

Yip!'s syntax is whitespace dependent. For example, a colon after a word makes a command; a colon before a word makes a symbol; a colon alone opens a definition.

Yip! is concatenative

In Yip!, you write values and operations from left to right, and values sit on a work stack until a word picks them up and uses them.

1 2 + . # => 3

1 and 2 are pushed onto the work stack. + consumes them, adds their values, and pushes the result onto the work stack. Finally, . prints the result.

Various words to shuffle the stack are provided, some of them include:

dup  ( a -- a a )  : Copy the top value in the work stack.
drop ( a -- )      : Discard the top value in the work stack.
swap ( a b -- b a) : Swap the top two values in the work stack.

Definitions

You can name a sequence of operations using :. This is what we call a word:

: square \( n -- n^2 ) dup * ;
5 square . # => 25

dup copies the top value in the work stack. * consumes them, multiplies them together, and pushes the result onto the work stack. Calling the word square is the same as if we were calling the words that make up its definition individually.

We can also give a name to a sequence of operations as a value, by using :=.

:= answer 40 2 + ;

answer . # => 42

Note: := only looks for the top of the work stack. If the body leaves more than one value in the stack those extra values remain on the work stack. If the body leaves no values in the stack, an error will be raised.

Quotations

Yip! also lets you treat pieces of code as values. A quotation is an excerpt of code between brackets, and you can call them using the call word.

# call : ( quot -- ... )

[ 1 2 + ] call . # => 3
1 [ 2 + ] call . # => 3
1 2 [ + ] call . # => 3

Composing and currying quotations

Yip! provides the words curry and compose for composing values and quotations:

A value is callable if it is a quotation, a curry object, or a compose object.

Combinators

Words that take quotations (or other callables) as arguments are commonly called combinators. Like its predecessors in concatenative programming, Yip! uses combinators extensively: things like iteration, callbacks, conditionals, are all implemented using quotations.

Dataflow combinators

Dataflow combinators express common dataflow patterns.

One important dataflow combinator is dip, with a stack effect of ( x quot -- x ), which pops a value from the work stack, executes a quotation, and then pushes that value back into the work stack:

"hello" "world" [ . ] dip . # => "hello" "world"

Yip! also borrows various dataflow combinators from Factor, such as

# Using `bi' to get two things out of one value
9 [ 1 fx+ ] [ 1 fx- ] bi swap . .         #=> 10 8

# Using `bi*' to do two different things to two values
"hello" 10 [ " world" .concat ] [ 2 fx* ] 
swap . .                                  #=> 20 "hello world"

# Using `bi@` to do one thing to two values
"hello" 10 [ type-of ] bi@ swap . .       #=> :string :fixnum

Named stacks

The work stack is not the only stack we can use, Yip! provides the programmer with an open-ended family of named stacks and operations over them:

10 >a        # Push `10' from the work stack to the stack `a'

             # Peek the top of the stack `a'
a. .         # => 10

a> drop      # Pop from the stack `a' into the work stack

10 >b
b/           # Discard the top of the stack `b'

20 >a 
             # Return whether the stack has values in it
a~ .         # => true
b~ .         # => false

These operations are named stash, peek, unstash, slash and check respectively.

Command syntax

Yip! has a syntax sugar called command syntax, a word suffixed by a colon is deferred until a semicolon or exclamation mark is seen, and only then it is expanded:

# The following command blocks
word: a b c ;
word: a b c!

# ...are desugared to the following
a b c word

Command blocks can nest. An exclamation mark is equivalent to a semicolon, and is provided for aesthetic reasons when writing code.

Yip! is object-oriented

Objects respond to messages. Messages are written by prefixing a word with a period:

{:greeting "Hello!"} .greeting . # => "Hello!"

Sending the message greeting asks the object how it responds to that selector. In this case, it produces the string "Hello!".

Yip! has two forms of dispatch:

Sending a message to an object goes as follows:

  1. Resolve the selector according to prototype dispatch.
  2. If the resolved value is callable, push the receiver onto the work stack and call it.
  3. Otherwise, push the resolved value onto the work stack.

Command syntax can also be used for messages:

# The following command block
.word: a b c ;

# ...is desugared to
[ a b c ] dip .word

You can use the object/new word to create an empty object.

Prototyping

Yip! does not separate classes from instances. Every value that participates in object dispatch is an object, and any object can serve as another's prototype.

Given the following object:

:= Wolf
  object/new .tap: [
    : speak \( self -- ) drop "awoo!" . ;
  ]! ;

We can use .derive to create a new object whose prototype is this object:

:= Coyote Wolf .derive ;

We can then add our own methods, or override inherited methods as we would with any object:

Coyote .with: [
  : speak \( self -- ) drop "yip yap yop" . ;
] ;

Coyote .speak # => yip yap yop

When Yip! sends speak to Coyote, it searches:

  1. Coyote's own slots
  2. Wolf's slots
  3. Wolf's prototype
  4. and so on...

Objects in practice

Everything is an object in Yip!, even primitive values like booleans, numbers, and strings. Like user-defined objects, every primitive type participates in prototype dispatch through a prototype object residing in the Lobby.

Conditionals are combinators defined in the Boolean prototype:

true .when: [ "Hello, world!" . ]; # => "Hello, world!"
true .unless: [ "I am evil!" . ];  # => <no output>
false .if: [
  "True" .
] [
  "False" .
];                                 # => "False"

Arrays and strings implement a sequence protocol, so you can index them (strings are indexed byte-wise), and iterate through them by sending messages to them:

"hello" .nth: 0! .      # => 104

(1 2 3 4) .each: [ . ]! # => 1 2 3 4

What is the focus?

When Yip! evaluates code, there is somewhere it is evaluating from.

That "somewhere" is the current focus, the object whose context is currently being used. Bare words are looked up in it, and definitions while an object is focused become part of the focused object.

We can focus a value, consuming it from the work stack, using the focus word, and get it back into the work stack once we're done by using unfocus:

object/new focus
: speak drop "Foobar!" . ;
unfocus .speak # => "Foobar!"

Yip! provides two combinators for objects to manipulate focus, as it is a common operation for Yip! programs: .with and .tap.

.with and .tap do the sameer focus-call-then-unfocus dance we did manually in the previous example, but .with drops the value after unfocusing it, while .tap keeps it in the work stack.

We can rewrite the previous example using .tap like this:

object/new .tap: [
  : speak drop "Foobar!" . ;
]! .speak # => "Foobar!"

Read this as:

  1. create an object
  2. temporarily become that object
  3. define speak-phrase and speak on it
  4. un-become that object, leaving it in the work stack
  5. send the message speak to it, executing the method defined inside of it

Yip! also provides the syntax sugar .[ ... ] as a spelling of .with for smaller, inline usage:

# The following syntax
object .[ ... ]

# ...is desugared to the following
object focus ... unfocus drop

Various words to interact with the current focus are provided, many of them providing the bridge between focus dispatch and object dispatch. Some of them include:

add-slot  ( name value -- ) : Add a slot to the current focus
send      ( selector -- )   : Send a message to the current focus

Focus is a stack

A program can enter another context while already inside one. Yip! represents these nested contexts as an ordered focus stack:

│ [current focus]
│ [parent focus]
│ [...]
▼ [Lobby]

When Yip! needs to resolve a word, it starts at the current focus, proceeding towards its parent foci if it can't find the word in the current focus. When a value is unfocused, Yip! removes it and resumes with the previous one as the current focus.

Focus dispatch is independent of prototype dispatch. Focus determines how bare words are resolved, while prototypes determine how messages are resolved.

The root object in the focus stack is called the Lobby. This is where primitives like object/new live.

The ambient object

Pushing to a named stack does not act in the current focus by default unless the slot named already contains a stack, so the focus stack contains an ambient object in its base, that cannot be seen or moved, where named stacks end up by default.

│ [current focus]
│ [parent focus]
│ [...]
│ [Lobby]
▼ [ambient object]

Forcing a slot to be promoted to a stack (or created as one) in the current focus if the slot named contains a scalar value can be done using the >> sigil:

object/new .tap: [
  1 >a
  2 >a
];

.[ a> a> . . ] # => 2 1

Lifting

Since mentioning a slot calls it if its callable, Yip! provides an operation called lifting to get the value of a slot no matter its type. It can be used with the ^ prefix.

object/new .tap: [
  : my-word "Hello!" display nl ;
  
  my-word    # => Hello!
  ^my-word . # => [ "Hello!" display nl ]
];

Objects are multistacks

An object is a collection of named slots, with each slot holding either a value or a stack of values. The current focus tells Yip! which object's slots to use for named stack operations:

object/new .tap: [ 10 >>a ];
object/new .tap: [ 20 >>a ];

# Both objects hold a slot named `a` which, when the object is focused, acts
# as a named stack:

.[ a> . ] # => 20
.[ a> . ] # => 10

Named stacks in focus resolution

When an slot either replies to a message or is found via focus resolution and is a stack, the value at the top of it is used as its value for lookup purposes:

object/new .tap: [
  : my-word "Hello!" display nl ;
  
  my-word # => Hello!
  [ "Goodbye!" display nl ] >>my-word
  my-word # => Goodbye!
  my-word/
  my-word # => Hello!
];

To be continued...

:site use render-self: {:page-title "A tour through Yip!"};