Cooler Ranch

Primer on Ports

2017-04-05T11:15:00+00:00

Below is a writeup from the April 2017 Houston Elixir Meetup about using ports to talk to the world outside of the BEAM. Examples and slides can be found here.

Primer on Ports

Why ports?

Ports are the safest way to communicate to external programs outside of the BEAM that are local to the machine.

It can be a large time saver to re-use a solution implemented in another language instead of writing a solution from scratch in Elixir. Or possibly, Elixir isn’t always the best fit for a certain use case, but may be useful for coordination, fault-tolerance, and glue between multiple programs.

A port is owned by some process and can be communicated to via message passing, very similarly to any other process. A port will communicate to an external process on your operating system through STDIN and STDOUT (by default). Because of this, it’s best to favor passing large work loads to the external program and return information instead of a chatty implementation with smaller computations.

Ports are OTP compliant, so if you have a port open to a network socket or another application and you tear down part of your supervision structure, the port will be closed gracefully (on the Erlang side, more about that below).

Usage

Elixir has a Port module, which is merely a thin wrapper around Erlang’s port BIFs.

To open a port, simply use the command Port.open/2:

# Extremely quick demo
iex(11)> port = Port.open({:spawn, "whoami"}, [:binary])
#Port<0.1267>
iex(12)> flush()
{#Port<0.1267>, {:data, "geoff\n"}}
:ok
iex(13)> Port.info port
nil

# Open a port
iex(14)> cat_port = Port.open({:spawn, "cat"}, [:binary])
#Port<0.1268>

# Sending messages
iex(15)> send cat_port, {self(), {:command, "Hello from port!"}}
{#PID<0.80.0>, {:command, "Hello from port!"}}
iex(16)> send cat_port, {self(), {:command, "Goodbye from port!"}}
{#PID<0.80.0>, {:command, "Goodbye from port!"}}

# What's in our mailbox? Why it's messages from our port!
iex(17)> flush()
{#Port<0.1268>, {:data, "Hello from port!"}}
{#Port<0.1268>, {:data, "Goodbye from port!"}}
:ok

# Info on an open port
iex(18)> Port.info cat_port
[name: 'cat', links: [#PID<0.80.0>], id: 10144, connected: #PID<0.80.0>,
 input: 34, output: 34, os_pid: 2064]
iex(19)> send cat_port, {self(), :close}

# Close the port
{#PID<0.80.0>, :close}

# Confirm what we got in the mailbox
iex(20)> flush()
{#Port<0.1268>, :closed}
:ok

# Calling Port.info/1 on a closed port returns nil
iex(21)> Port.info cat_port
nil

Other functions

Be sure to close ports when you’re done with them with Port.close/1.
If the external progam exits, the port closes. If a port is closed, the function Port.info/1 will return nil.
Ports communicate through sending 2-tuples in the form of {:command, command} and {:data, data}, and can be reached with the send/2 function from Elixir’s Kernel module or from Port.command/3.
:spawn and :spawn_executable are the two settings you will typically use when opening a port.
:spawn_executable is more strict than :spawn, and requires a full file path. :spawn will do some favors looking in your $PATH environment, but it has some limitations.
The process that owns the port connection can transfer its ownership to another process by sending {self(), {:connect, new_pid} to the port.
There are many more settings explained in the Erlang Docs.

Aside on `:spawn_driver` and `:fd`:

The :spawn_driver and :fd options are for Deep Wizardry, only to be used with extreme caution and for Very Good Reasons.

:spawn_driver is for linked-in drivers, just like ports but inside the BEAM and (typically) implemented in C. This comes with the added risk of crashing the VM Hooray!
:fd is used for for file descriptors used by the VM, and should be used only for re-implementing parts of the VM.

Implementation

It’s generally a good practice to wrap port operations in a GenServer. When doing so, there are a few things to add to the GenServer:

Code for opening the port.

def init(_) do
    port = Port.open({:spawn, "cmd args"}, [:binary])
    ## do some other stuff if necessary
    {:ok, {initial_state, port}}
  end

I like to include this in the GenServer’s init function if you’re going to have a relationship between your process and external program. This way, processes can not send your GenServer any messages until the port is up and running.

You would probably want to carry this port identifier with the GenServer’s State, mostly so you can distinguish between ports and ensure youre GenServer doesn’t act on similar looking messages from elsewhere.

Code for listening/receiving messages on the port:

  def handle_info({port, {:data, payload}}, {_old_state, port}) do
    ## do some work here
    {:noreply, {new_state, port}}
  end

You can view the the /port_demo directory for a quick implementation of a port wrapped in a GenServer. The key stuff here is included in top_server.ex. While this is a contrived example, it should give the basics on how this work on the Elixir/Erlang side.

This is a quick example of sending/receiving messages on the elixir side, and does not include full error handling.

Note:

This is a contrived example of wrapping a Unix OS program in a port. If you are simply getting information from a program and reporting back its return result, please use System.cmd/3 instead. You will save yourself a lot of time. This function also uses ports to fetch information, and it’s worthwhile to look at the implementation.

Best practices, tips, and thoughts

Wrap your port operations in a GenServer to offer a clean API to use the external service within Elixir.
You can communicate to a port on [3] and [4] instead of STDIN/STDOUT by setting :nouse_stio in the list of settings to Port.open/2. This is helpful if you want to print information to STDIN/STOUT and not have Erlang catch all of printed messages (say, for debugging).
If you don’t know how your external program will close down, it’s a good practice to wrap the command in a bash script. Below is the example from Elixir’s recommendation on preventing Zombie Processes.

#!/bin/sh
"$@"
pid=$!
while read line ; do
  :
done
kill -KILL $pid

The data passed between the two programs will be strings or lists of bytes, but can be serialized through something like JSON or whatever lowest common denomenator you have chosen for your project. Additionally, there are language-specific libraries that can convert Erlang terms within the program so you can send/receive messages with :erlang.term_to_binary/1 and :erlang.binary_to_term/1.

References / Further Reading

Statisaur: Elixir Library for Univariate Statistics

2017-02-01T13:30:00+00:00

Using Dialyzer in Elixir

2016-12-07T13:26:00+00:00

Below is the general presentation from the December 2016 Houston Elixir Meetup on strings and binaries.

DialyzerDemo

Typing in Erlang

Erlang (and by Elixir) is a dynamically typed language. As a dynamic language, some type errors cannot by cought until runtime. Some parts of the language inherently make type errors harder to get than other dynamic languages: compilation, pattern matching, guard clauses, and the approach to errors go a long way. However, it does not grant the absolute type safety that you might get from ML-family languages or even languages that are just statically typed.

What is dialyzer?

The Dialyzer, a DIscrepancy AnalYZer for ERlang programs. Dialyzer is a static analysis tool used for checking types and other discrepancies such as dead or unreachable code.

Dialyzer examines .erl or .beam files (not .ex or .exs) – more on this in a minute.

Success typing

Success typing takea a very optomistic treatment of your code. Dialyzer will only emit errors when it can guarantee there is a problem (no false positives). While problematic code can still pass through the analysis, if dialyzer says there is a problem, there definitely is a problem.

As Dialyzer examines your code, it develops constraints on your functions. Dialyzer will assume your functions are being used properly until it can prove that the function will always be wrong.

Example: Boolean `and` function

The and function in a static language might look something like this:

and :: Bool -> Bool -> Bool
and x y | x == True && y == True = True
        | otherwise = False

In this static language, the compiler would throw and error at you if you used the function with anything other than a boolean value for both arguments.

An Elixir variant may look like this:

defmodule BooleanFuns do
   def and(true, true) do
    true
  end
  
  def and(false, _) do
    false
  end
  
  def and(_, false) do
    false
  end 
end

This code would be too uncertain for a static language to compile, but is A-OK in Elixir. Dialyzer is fine with it, too!

Constraint generation occurs with functions that it knows can only be of a certain type. For example, using the + is enough information for dialyzer to require this function receive only numbers. Guard clauses like is_atom or is_list will also develop more constraints for your functions.

Once the constraints are generated, it is time to solve them, just like a puzzle. The solution to the puzzle is the success typing of the function. Conversely, if no solution is found, the constraints are unsatisfiable and you have a type violation on your hands.

Using dialyzer

Dialyzer comes with current distributions of Erlang. If you have Elixir, you already have dialyzer.

Dialyzer uses a Persistent Lookup Table (PLT) to keep track of all of your constraints. As it analyzes your code, it includes more information onto the table.

The PLT can be built with the following command: dialyzer plt Keep in mind that the PLT takes a while to build, but only needs to be performed once.

Dialyzer can only work on .erl and .beam code, so we have to compile our code in order to use the dialyzer on Elixir code.

Using dialyxir

Remembering to compile each time we want to use dialyzer is a chore. Luckily, the dialyxir library includes handy mix tasks to compile and do other favors for us when we use dialyzer. After including dialyxir as a dependency in mix.exs, pulling the dependency, and building the PLT, you can run dialyzer through mix:

mix dialyzer

Alternatively, you can install dialyxir globally on your machine.

git clone https://github.com/jeremyjh/dialyxir
cd dialyxir
MIX_ENV=prod mix do compile, archive.build, archive.install

I recommend doing this, so you can use it on any project without adding it to the development dependencies.

Tip

Remember, you can always check types in iex with the i function:

iex(2)> i "elixir"
Term
  "elixir"
Data type
  BitString
Byte size
  6
Description
  This is a string: a UTF-8 encoded binary. It's printed surrounded by
  "double quotes" because all UTF-8 encoded codepoints in it are printable.
Raw representation
  <<101, 108, 105, 120, 105, 114>>
Reference modules
  String, :binary
iex(3)> i 'elixir'
Term
  'elixir'
Data type
  List
Description
  This is a list of integers that is printed as a sequence of characters
  delimited by single quotes because all the integers in it represent valid
  ASCII characters. Conventionally, such lists of integers are referred to as
  "charlists" (more precisely, a charlist is a list of Unicode codepoints,
  and ASCII is a subset of Unicode).
Raw representation
  [101, 108, 105, 120, 105, 114]
Reference modules
  List

Using @specs

Dialyzer doesn’t require typespecs to run and catch errors. However, the @specs act as hints for the program and can help it with developing better constraints. The @spec flag will tell dialyzer to include this information in the PLT.

Sources:

Dialyzer documentation

Dialyxir on Hex.pm

Learn You Some Erlang: Fred Hebert

Little Elixir & OTP Guidebook: Benjamin Tan

Success Typing Paper: Tobias Lindahl & Konstantinos Sagonas

Binaries and Strings in Elixir

2016-11-02T13:12:00+00:00

Below is the general presentation from the November 2016 Houston Elixir Meetup on strings and binaries. It synthesizes a lot of sources about strings on the BEAM, but provides a quick and dirty guide to how strings/binaries work on this platform.

The talk was given days after Halloween, so we indulged in spooky examples.

Binaries and strings

What are binaries?

Strings are UTF-8 Encoded binaries.

Binaries are a sequence of bytes, denoted between the << and >> angle brackets.

A Binary is a bitstring where the number of bits is divisible by 8.

Binaries that are valid UTF - 8 will be represented as strings.

Strings have a data structure and a printable representation.

  iex> <<71, 45, 71, 45, 71, 45, 71, 72, 79, 83, 84>>
  "G-G-G-GHOST"

This example shows when you enter the codepoints into IEX, you receive a representation as a string.

You can also get a character’s codepoint by using the ? operator.

  iex> ?👻
  128123

Elixir also supports a shorthand for referring to unicode characters:

  iex> "\u0068"
  "h"

Graphemes vs. codepoints

The data structure is composed of codepoints, which can combine to create individual graphemes. In unicode, some codepoints are used to modify other characters, like é.

  iex> string = "\u0065\u0301"
  "é"
  iex> String.codepoints string
  ["e", "́"]

Keep in mind that in order to know a string’s length (String.length/1, Elixir will have to traverse this sequence of bytes to return the number of printable graphemes from your data (and O(N) operation). You can also use Kernel.byte_size/1 to know the size of a given binary, which is an O(1) operation.

  iex> byte_size("é")
  2
  iex> String.length("é")
  1

There be dragons 🐉

Some characters can be represented as one codepoint or as a combination of multiple codepoints:

  iex> {combined, individual} = {"e\u0308", "\u00EB"}
  {"ë", "ë"}

  iex> combined == individual
  false

  iex> String.equivalent?(combined, individual)
  true

Charlists

Charlists are typically used for compatability reasons with Erlang. They are represented as single-quoted strings.

  iex> "Spooky scary skeletons" |> String.to_charlist
  'Spooky scary skeletons'

One advantage/feature of this is that you can iterate over the sequence with functions you would use to work with lists.

  iex> spooky_charlist = 'Spooky scary skeletons'
  'Spooky scary skeletons'

  iex> hd spooky_charlist
  83

  iex> <<83>>
  "S"

  iex> spooky_string = "Spooky scary skeletons"
  "Spooky scary skeletons"

  iex> hd spooky_string
  ** (ArgumentError) argument error
      :erlang.hd("Spooky scary skeletons")

  iex> is_list spooky_string
  false

  iex> is_list spooky_charlist
  true

Large binary space

Large binaries (> 64 bytes) stored in separate area from a process’ heap.

This allows for fast message passing as only pointer sent between processes

Can save a lot of memory as well (no needless copying)

Can be long delay before being reclaimed by GC. The large binary has a counter of all of the other processes that are using it. In order to be garbage collected, all processes which have “seen” the binary must first do a garbage collection.

If garbage collection does not happen often enough, the large binaries can grow and crash system.

If your process is only referencing a tiny part of a large binary, it might be worthwhile to use :binary.copy to explicitly copy the portion you need from a large binary so that the large binary can be garbage collected sooner. More info in erlang’s docs.

See also Evan Miller’s Template of Doom post.

IO Lists

Concatenating strings costs memory to compose items into a single string. Since our data is immutable, it can be cheaper and faster to work with lists of references that can easily be appended.

IO lists allow you to create sequences of binaries and other IO lists, which will ultimately be flattened when the list is written (to disk for example). This has advantages because that final string never requires memory within your application. It only exists in the destination. Additionally, IO lists work with lists have have layers of nesting.

  iex> first = "David "
  "David "
  iex> mi = "S. "
  "S. "
  iex(41)> last = "Pumpkins"
  "Pumpkins"
  iex> IO.puts [[first, [mi]], last]
  David S. Pumpkins
  :ok

Additional Sources / Further Reading

Elixir and Unicode, Part 2: Working with Unicode Strings from Nathan Long / Big Nerd Ranch
Hitchhiker’s Guide to the BEAM Lecture from Robert Virding, Includes information on the large binary space on the BEAM, and lots of other information about the VM.
Erlang’s binary manpage
String Module from Elixir.
Template of Doom from Evan Miller

Up and Running in Elixir

2016-10-05T13:34:00+00:00

Below is the general presentation from the December 2016 Houston Elixir Meetup on writing code that can run and be distributed.

Runnable Code

Includes some of the most common ways to run elixir code in your development environment, and ways of creating OTP Releases.

Outline:

Compile from command line / REPL
Running Elixir Script (.exs)
Compile from Mix
Application behaviour
Escript
Releases & Distillery
Final Thoughts, Caveats

Compile From Command Line/REPL

The REPL / iex is a great place to test out expressions and general functionality of a module in real time. If you have a file in Elixir that you want to compile and load into the REPL, use the c/1 command. We’re using DemoModule, a contrived example.

$ iex
Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Interactive Elixir (1.3.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> c "demo_module.ex"
[DemoModule]
iex(2)> DemoModule.say_it
Hello from Elixir.DemoModule
:ok

Additionally, the user can keep the same iex session after making changes to code. Use command recompile/0 from iex.

Running Elixir Script (.exs)

Elixir can run in an interpreted form. These files have the extension .exs to differentiate from compilable Elixir code. .exs files are typically used for configuration. These files can be exectured directly from the command line by calling on Elixir to run, or by loading it into iex. Note that loading the script into iex will execute the script eagerly.

$ elixir demo_script.exs
Hello from an Elixir script!

iex(1)> c "demo_script.exs"
Hello from an Elixir script!
[]
iex(2)>

Code example of script here.

Using Mix

Once your codebase expands beyond a few modules, or you start to include dependencies, you will want to start using a tool to manage the code. Elixir ships with Mix, which functions as a test runner, dependency manager, and a build tool.

Make a new project with $ mix new app_name. This will create a directory structure, along with config files, a place for your dependencies and beam files to live, and a test directory. View module examples here.

You can even start iex sessions and let mix compile and load in your modules and listed dependencies:

$ iex -S mix
Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Compiling 1 file (.ex)
Generated with_mix app
Interactive Elixir (1.3.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> WithMix.say_it
Hello from Elixir.WithMix
:ok

Detour: Using the Application Behaviour

The Application behaviour is used typically in a module that exposes a public-facing API. Application is responsible for starting your supervision trees, and calling any cleanup code when your application exits. Example of directory and modules here.

The new application module looks like this (demo_application.ex):

defmodule DemoApplication do
  alias DemoApplication.{DemoSupervisor, DemoModule}
  use Application 

  def start(_type, _args) do
    DemoSupervisor.start_link()
  end

  def use_module() do
    DemoModule.say_it(:demo_module)
  end
end

You would also specify this module in the application/0 function in your mix.exs file:

  def application do
    [
      applications: [:logger],
      mod: {DemoApplication, []}
    ]
  end

Mix will use the start/2 callback in the module you specify, along with any arguments specified in the second element of the tuple.

All subsequent examples will use both Mix and the Application behaviour.

Escripts

Generates an executable from BEAM files. Requires the user already have the VM installed.

To use an Escript, you will need a separate module to run the code from the script. We’ll create a module called demo_CLI.ex in our lib directory. It requires a function called main that accepts arguments from the command line. The module looks like this:

defmodule DemoApplication.CLI do
  alias DemoApplication.DemoModule

  def main(args) do
    args |> parse_args() |> do_process()
  end

  def parse_args(args) do
    options = OptionParser.parse(args, aliases: [h: :help])

    case options do
      {[help: true], _, _} -> :help
      _ -> :run
    end
  end

  def do_process(:run) do
    DemoModule.say_it(:demo_module)
  end

  def do_process(:help) do
    IO.puts """
    Welcome to Demo App.

    """
  end
end

In mix.exs, include escript: escript within the project/0 function and create a new escript/0 function:

def escript, do: [main_module: HouTax.CLI]

Then compile the script:

$ mix escript.build
Compiling 4 files (.ex)
Generated demo_application app
Generated escript demo_application with MIX_ENV=dev

This creates an executable called demo_application. You can run it immediately with ./demo_application. View full directory and module here.

Releases with Distillery

Distillery is a library by Paul Schoenfelder (bitwalker), and is a ground-up rewrite/replacement of his popular repository ExRM (Elixir Release Manager). Distillery autmoates a lot of the process to generate OTP releases.

An OTP release includes your compiled BEAM files, as well as some form of the Erlang run time system (ERTS/erts). Should be compiled on the target OS and CPU architecture. Optionally, you can elect to produce the release without the runtime, but you will need to ensure Erlang is installed on the targeted host machine.

We’ll go through the minimum here, but the documentation is very thorough.

Example project here. Include distillery in your mixfile’s dependencies:

  defp deps do
    [{:distillery, "~> 0.10.0"}]
  end

and fetch the dependencies with $ mix deps.get && mix compile

Then, run mix release.init to generate a new /rel directory with a config file inside.

At this point, you can edit the config file before making the final release.

Run mix release to generate you new release: For Development:

$ mix release
==> Assembling release..
==> Building release demo_application:0.1.0 using environment dev
==> You have set dev_mode to true, skipping archival phase
==> Release successfully built!
    You can run it in one of the following ways:
      Interactive: rel/demo_application/bin/demo_application console
      Foreground: rel/demo_application/bin/demo_application foreground
      Daemon: rel/demo_application/bin/demo_application start

For Production:

$ mix release --env=prod
==> Assembling release..
==> Building release demo_application:0.1.0 using environment prod
==> Including ERTS 8.1 from /usr/local/Cellar/erlang/19.1/lib/erlang/erts-8.1
==> Packaging release..
==> Release successfully built!
    You can run it in one of the following ways:
      Interactive: rel/demo_application/bin/demo_application console
      Foreground: rel/demo_application/bin/demo_application foreground
      Daemon: rel/demo_application/bin/demo_application start

The production variant will produce a gzipped tarball that you can deploy on your target machine.

Final Thoughts, Caveats

Thanks for taking the ride on this tour of creating executable files in Elixir. As you can see, there are a number of options of generating code from quick throwaway scripts to full applications that can be run on machines that don’t even have Erlang installed.

This is a surface-level view of the mosst common ways you can get your code into an executable state or out into the world. However, there is much much more to using Elixir/Erlang including running multiple nodes, running an application in the background with mix in --detached mode, hot upgrades with releases, and tuning the VM (like raising the maximum number of processes).

Useful sources:

Programming Elixir (Dave Thomas, Pragmatic Bookshelf) Chapter 1
Elixir in Action (Sasa Juric, Manning) Chapters 11-13 are probably the most useful for using mix, the Application behaviour, escripts, and releases.
Distillery Documentation
Elixir-Lang.org’s Getting Started Guide

Houston Elixir Meetup

2016-09-15T13:30:00+00:00

Founded in September of 2016 with crertel, the Houston Elixir Meetup provides the Houston developer community a resource for discussing Elixir, Erlang, the underlying virtual machine and related programming topics and concepts.

You can join the community through the group’s Meetup Page, or view prior projects on our Github profile.

HouTax: ETL and OTP Demo

2016-08-21T13:30:00+00:00

HouTax - Demo. This is a simple ETL exercise leveraging OTP.

Gibran: Natural Language Processing in Elixir

2016-08-16T13:30:00+00:00

Cooler Ranch

Primer on Ports

Primer on Ports

Why ports?

Usage

Other functions

Aside on :spawn_driver and :fd:

Implementation

Note:

Best practices, tips, and thoughts

References / Further Reading

Statisaur: Elixir Library for Univariate Statistics

Using Dialyzer in Elixir

DialyzerDemo

Typing in Erlang

What is dialyzer?

Success typing

Example: Boolean and function

Using dialyzer

Using dialyxir

Tip

Using @specs

Sources:

Binaries and Strings in Elixir

Binaries and strings

What are binaries?

Graphemes vs. codepoints

There be dragons 🐉

Charlists

Large binary space

IO Lists

Additional Sources / Further Reading

Up and Running in Elixir

Runnable Code

Outline:

Compile From Command Line/REPL

Running Elixir Script (.exs)

Using Mix

Detour: Using the Application Behaviour

Escripts

Releases with Distillery

Final Thoughts, Caveats

Useful sources:

Houston Elixir Meetup

HouTax: ETL and OTP Demo

Gibran: Natural Language Processing in Elixir

Aside on `:spawn_driver` and `:fd`:

Example: Boolean `and` function