User Guide

The combinators fit very well to express intent of communication behavior. It gives rich abstractions to hide the networking complexity and help us to compose a chain of network operations and represent them as pure computation, building new things from small reusable elements.

assay-it is implemented over ᵍ🆄🆁🅻 library - a “combinator” library for network I/O. Combinators open up an opportunity to depict computation problems in terms of fundamental elements like physics talks about universe in terms of particles. The only definite purpose of combinators are building blocks for composition of “atomic” functions into computational structures. assay-it uses a powerful symbolic expressions of combinators to implement declarative language for testing suite development.

Table of contents

  1. User Guide
    1. Background
      1. Combinators
      2. High-order functions, how to compose protocol primitives
      3. Combinators and its syntax
    2. Writer combinators
      1. Method
      2. Target URI
      3. Query Params
      4. Request Headers
      5. Request payload
    3. Reader combinators
      1. Status Code
      2. Response Headers
      3. Response Payload
      4. Assert Payload
    4. Chain networking I/O

Background

Combinators

Let’s formalize principles that help us to define our own abstraction applicable in functional programming through composition. The composition becomes a fundamental operation: the codomain of 𝒇 be the domain of 𝒈 so that the composite operation 𝒇 ◦ 𝒈 is defined. Our formalism uses Arrow: IO ⟼ IO as a key abstraction of networking combinators.

// Arrow: IO ⟼ IO
type Arrow func(*Context) error

It is a pure function that takes an abstraction of the protocol context and applies morphism as an “invisible” side-effect of the composition.

Following the Input/Process/Output protocols paradigm, the two classes of combinators are defined:

  • The first class is writer (emitter) morphism combinators, denoted by the symbol ø across this guide and example code. It focuses inside the protocol stack and reshapes requests. In the context of HTTP protocol, the writer morphism is used to declare HTTP method, destination URL, request headers and payload.
  • Second one is reader (matcher) morphism combinators, denoted by the symbol ƒ. It focuses on the side-effects of the protocol stack. The reader morphism is a pattern matcher, and is used to match response code, headers and response payload, etc. Its major property is “fail fast” with error if the received value does not match the expected pattern.

High-order functions, how to compose protocol primitives

Arrow can be composed with another Arrow into new Arrow and so on. Only product “and-then” composition style is supported. It builds a strict product Arrow: A ◦ B ◦ C ◦ ... ⟼ D. The product type takes a protocol context and applies “morphism” sequentially unless some step fails. Use variadic function http.Join, http.GET, http.POST, http.PUT and so on to compose HTTP primitives:

// Join composes HTTP arrows to high-order function
// (a ⟼ b, b ⟼ c, c ⟼ d) ⤇ a ⟼ d
func http.Join(arrows ...http.Arrow) http.Arrow

//
var a: http.Arrow = /* ... */
var b: http.Arrow = /* ... */
var c: http.Arrow = /* ... */

d := http.Join(a, b, c)

Ease of the composition is one of major intent why syntax deviates from standard Golang HTTP interface. http.Join produces instances of higher order http.Arrow type, which is composable “promises” of HTTP I/O and so on. Essentially, the network I/O is just a set of Arrow functions. These rules of Arrow composition allow anyone to build a complex HTTP I/O scenario from a small reusable block.

Combinators and its syntax

The combinator domain specific language consists of multiple packages, import them all into Golang module

import (
  // context of http protocol stack
  "github.com/fogfish/gurl/http"

  // Writer (emitter) morphism combinators. It focuses inside the protocol stack
  // and reshapes requests. In the context of HTTP protocol, the writer morphism
  // is used to declare HTTP method, destination URL, request headers and payload.
  // single letter symbol (e.g. ø) makes the code less verbose
  ø "github.com/fogfish/gurl/http/send"

  // Reader (matcher) morphism combinators. It focuses on the side-effects of
  // the protocol stack. The reader morphism is a pattern matcher, and is used
  // to match response code, headers and response payload, etc. Its major
  // property is “fail fast” with error if the received value does not match
  // the expected pattern. 
  // single letter alias (e.g. ƒ) makes the code less verbose
  ƒ "github.com/fogfish/gurl/http/recv"
)

func TestWebSiteOnline() http.Arrow { /* ... */ }

Writer combinators

Writer (emitter) morphism combinators. It focuses inside the protocol stack and reshapes requests. In the context of HTTP protocol, the writer morphism is used to declare HTTP method, destination URL, request headers and payload.

Method

Use http.GET(/* ... */) combinator to declare the verb of HTTP request. The language declares a combinator for most of HTTP verbs: http.GET, http.HEAD, http.POST, http.PUT, http.DELETE and http.PATCH.

func TestGetXxx() http.Arrow {
  return http.GET(/* ... */)
}

func TestPutXxx() http.Arrow {
  return http.PUT(/* ... */)
}

Use ø.Method combinator to declare other verbs

func TestXxx() http.Arrow {
  return http.Join(
    ø.Method("OPTIONS"),
    /* ... */)
}

Target URI

Use ø.URI(string) combinator to specifies target URI for HTTP request. The combinator uses absolute URI to specify protocol, target host and path of the endpoint.

func TestXxx() http.Arrow {
  return http.GET(
    ø.URI("http://example.com"),
    /* ... */)
}

The ø.URI combinator is equivalent to fmt.Sprintf. It uses percent encoding to format and escape values.

http.GET(
  ø.URI("http://example.com/%s", "foo bar"),
)

// All path segments are escaped by default, use ø.Authority or ø.Path
// types to disable escaping

// BAD, DOES NOT WORK
http.GET(
  ø.URI("%s/%s", "http://example.com", "foo/bar"),
)

// GOOD, IT WORKS
http.GET(
  ø.URI("%s/%s", ø.Authority("http://example.com"), ø.PATH("foo/bar")),
)

Query Params

Use ø.Params(any) combinator to lifts the flat structure or individual values into query parameters of specified URI.

type MyParam struct {
  Site string `json:"site,omitempty"`
  Host string `json:"host,omitempty"`
}

func TestXxx() http.Arrow {
  return http.GET(
    /* ... */
    ø.Params(MyParam{Site: "example.com", Host: "127.1"}),
    /* ... */
  )
}

Use ø.Param to declare individual query parameters, this combinator is suitable for simple queries, where definition of dedicated type seen as an overhead

func TestXxx() http.Arrow {
  return http.GET(
    /* ... */
    ø.Param("site", "example.com"),
    ø.Param("host", "127.1"),
    /* ... */
  )
}

Request Headers

Use ø.Header[T any](string, T) to declares headers and its values into HTTP requests. The standard HTTP headers are accomplished by a dedicated combinator making it type safe and easy to use e.g. ø.ContentType.ApplicationJSON.

func TestXxx() http.Arrow {
  return http.GET(
    /* ... */
    ø.Header("Client", "curl/7.64.1"),
    ø.Authorization.Set("Bearer eyJhbGciOiJIU...adQssw5c"),
    ø.ContentType.ApplicationJSON,
    ø.Accept.JSON,
    /* ... */
  )
}

Request payload

Use ø.Send to transmits the payload to the destination URI. The combinator takes standard data types (e.g. maps, struct, etc) and encodes it to binary using Content-Type header as a hint. It fails if content type header is not defined or not supported by the library.

type MyType struct {
  Site string `json:"site,omitempty"`
  Host string `json:"host,omitempty"`
}

func TestSendJSON() http.Arrow {
  return http.GET(
    // ...
    ø.ContentType.JSON,
    ø.Send(MyType{Site: "example.com", Host: "127.1"}),
  )
}

func TestSendForm() http.Arrow {
  return http.GET(
    // ...
    ø.ContentType.Form,
    ø.Send(map[string]string{
      "site": "example.com",
      "host": "127.1",
    })
  )
}

func TestSendOctetStream() http.Arrow {
  return http.GET(
    // ...
    ø.ContentType.Form,
    ø.Send([]byte{"site=example.com&host=127.1"}),
  )
}

On top of the shown type, it also support a raw octet-stream payload presented after one of the following Golang types: string, *strings.Reader, []byte, *bytes.Buffer, *bytes.Reader, io.Reader and any arbitrary struct.

Reader combinators

Reader (matcher) morphism combinators. It focuses on the side-effects of the protocol stack. The reader morphism is a pattern matcher, and is used to match response code, headers and response payload, etc. Its major property is “fail fast” with error if the received value does not match the expected pattern.

Status Code

Use ƒ.Status.OK checks the code in HTTP response and fails with error if the status code does not match the expected one. Status code is only mandatory reader combinator to be declared. The all well-known HTTP status codes are accomplished by a dedicated combinator making it type safe (e.g. ƒ.Status is constant with all known HTTP status codes as combinators).

func TestXxx() http.Arrow {
  return http.GET(
    // ...
    ƒ.Status.OK,
  )
}

Sometime a multiple HTTP status codes has to be accepted ƒ.Code arrow is variadic function that does it

func TestXxx() http.Arrow {
  return http.GET(
    // ...
    ƒ.Code(http.StatusOK, http.StatusCreated, http.StatusAccepted),
  )
}

Response Headers

Use ƒ.Header combinator to matches the presence of HTTP header and its value in the response. The matching fails if the response is missing the header or its value does not correspond to the expected one. The standard HTTP headers are accomplished by a dedicated combinator making it type safe and easy to use e.g. ƒ.ContentType.ApplicationJSON.

func TestXxx() http.Arrow {
  return http.GET(
    // ...
    ƒ.Header("Content-Type", "application/json"),
    ƒ.Authorization.Is("Bearer eyJhbGciOiJIU...adQssw5c"),
    ƒ.ContentType.JSON,
    ƒ.Server.Any,
  )
}

The combinator support “lifting” of header value into the variable for the further usage in the application.

func TestXxx() http.Arrow {
  var (
    date time.Time
    mime string
    some string
  )

  return http.GET(
    // ...
    ƒ.Date.To(&date),
    ƒ.ContentType.To(&mime),
    ƒ.Header("X-Some", &some),
  )
}

Response Payload

Use ƒ.Body consumes payload from HTTP requests and decodes the value into the type associated with the lens using Content-Type header as a hint. It fails if the body cannot be consumed.

type MyType struct {
  Site string `json:"site,omitempty"`
  Host string `json:"host,omitempty"`
}

func TestXxx() http.Arrow {
  var data MyType

  return http.GET(
    // ...
    ƒ.Body(&data), // Note: pointer to data structure is required
  )
}

So far, utility support auto decoding of the following Content-Types into structs

  • application/json
  • application/x-www-form-urlencoded

For all other cases, there is ƒ.Bytes combinator that receives raw binaries.

func TestXxx() http.Arrow {
  var data []byte

  return http.GET(
    // ...
    ƒ.Bytes(&data), // Note: pointer to buffer is required
  )
}

Assert Payload

Combinators is not only about pure networking but also supports assertion of responses. Assert combinator aborts the evaluation of computation if expected value do not match the response. There are three type of asserts: type safe ƒ.Expect, loosely typed ƒ.Match and customer combinator.

Type safe: Use ƒ.Expect to define expected value as Golang struct. The combinator fails if received value do not strictly equals to expected one.

func TestXxx() http.Arrow {
  return http.GET(
    // ...
    ƒ.Expect(MyType{Site: "example.com", Host: "127.1"}),
  )
}

Loosely typed: Use ƒ.Match to define expected value as string pattern. In the contrast to type safe combinator, the combinator takes a valid JSON object as string. It matches only defined values and supports wildcard matching. For example:

// matches anything
`"_"`

// matches any object with key "site"
`{"site": "_"}`

// matches array of length 1 
`["_"]`

// matches any object with key "site" equal to  "example.com"
`{"site": "example.com"}`

// matches any array of length 2 with first object having the key 
`[{"site": "_"}, "_"]`

// matches nested objects
`{"site": {"host": "_"}}`

// and so on ...

func TestXxx() http.Arrow {
  return http.GET(
    // ...
    ƒ.Match(`{"site": "example.com", "host": "127.1"}`),
  )
}

Custom combinator: The type Arrow func(*http.Context) error is “open” interface to combine assert logic with networking I/O. These functions act as lense – focuses inside the structure, fetching values and asserts them. These helpers can do anything with the computation including its termination:

type MyType struct {
  Site string `json:"site,omitempty"`
  Host string `json:"host,omitempty"`
}

// a type receiver to assert the value
func (t *MyType) CheckValue(*http.Context) error {
  if t.Host != "127.1" {
    return fmt.Errorf("...")
  }

  return nil
}

func TestXxx() http.Arrow {
  var data MyType

  return http.GET(
    // ...
    ƒ.Recv(data),
    t.CheckValue,
  )
}

Chain networking I/O

Ease of the composition is one of major intent why combinators has been defined. http.Join produces instances of higher order combinator, which is composable into higher order constructs. Let’s consider an example where sequence of requests needs to be executed one after another (e.g. interaction with GitHub API):

// 1. declare a product type that depict the context of networking I/O. 
type State struct {
  Token AccessToken
  User  User
  Org   Org
}

// 2. declare collection of independent requests, each either reads or writes
// the context
func (s *State) FetchAccessToken() http.Arrow {
  return http.GET(
    // ...
    ƒ.Recv(&s.Token),              // writes access token to context
  )
}

func (s *State) FetchUser() error {
  return http.POST(
    ø.URI(/* ... */),
    ø.Authorization.Set(&s.Token), // reads access token from context
    // ...
    ƒ.Recv(&hof.User),               // writes user object to context
  )
}

func (s *State) FetchContribution() error {
  return http.POST(
    ø.URI(&s.User.Repos),          // reads user object from context
    ø.Authorization.Set(&s.Token), // reads access token from context
    // ...
    ƒ.Recv(&s.Org),                // writes user's contribution to context
  )
}

// 3. Composed sequence of requests into the chained sequence
func HighOrderFunction() (*State, http.Arrow) {
	var state State

	//
	// HoF combines HTTP requests to
	//  * https://httpbin.org/uuid
	//  * https://httpbin.org/post
	//
	// results of HTTP I/O is persisted in the internal state
	return &state, http.Join(
    state.FetchAccessToken(),
    state.FetchUser(),
    state.FetchContribution(),
	)
}

Hopefully you find it useful, and the docs easy to follow.

Feel free to create an issue if you find something that’s not clear and join our discussions to chat with other users and maintainers.