Embedding LuaJIT with FFI

Vadim Graboys
@deapthoughts
github.com/vadimg/luajit-talk

Who am I?

Working at Chartbeat for a year, mostly on a custom datastore written in C
Professionally programmed in C, C++, C#, Ruby, Python, Javascript (client-side and server-side), and now Lua
But for only 4 years total
Jack of all trades, master of none
This is also my first talk ever!

What's Lua and why would I want to put it in my binary?

Lua is small: implementation is < 20k lines of C
Lua is simple: only 1 datastructure
LuaJIT is a really, really fast Lua implementation
LuaJIT FFI is a really nice C/Lua interface

But also weird:
http://notebook.kulchenko.com/programming/lua-good-different-bad-and-ugly-parts

Lua Crash Course

Arrays are tables:

local arr = {1, 2, 3}

Dictionaries are tables:

local dict = {a = 1, b = 2}

Array indexes start at 1:

> arr[1]
1

Objects are also tables

Account = {
    balance = 0,
    new = function(self)
        local o = {}
        setmetatable(o, self)
        self.__index = self
        return o
    end,
    deposit = function(self, amt)
        self.balance = self.balance + amt
    end
}

local o = Account.new(Account)
o.deposit(o, 10)
print(o.balance)

Output: 10

Object syntatic sugar

Account = { balance = 0 }
function Account:new()
    local o = {}
    setmetatable(o, self)
    self.__index = self
    return o
end

function Account:deposit(amt)
    self.balance = self.balance + amt
end

local o = Account:new()
o:deposit(10)
print(o.balance)

Output: 10

Lua/C interface

The interface is an abstract stack

To send data or register functions to lua:

Push onto the stack
Tell lua to pop from the stack

To get data from lua:

Lua will push stuff onto the stack
Pop the data off the stack and convert to native C type

What's Chartbeat's problem?

Realtime analytics is hard.

130k HTTP requests/second (writes)
4k HTTP requests/second (reads)
130MB/second writes, 100MB/second reads
Reads are an aggregation of all data for a website
No datastore could perform well enough at a cost we liked, so we wrote our own

Memoryfly

Single-threaded, uses libevent
Stateful HTTP Server
Contains business logic
All in one: HTTP Server, Application Server, Database
Complex: only 2 people in the company can work on it
Fixed queries: adding a new query takes development time
Want to add ad-hoc query capability

Common query

What are the top k pages (filtered by some criteria)?

Similar to:
Given a list of strings, what are the top k most frequently occurring strings?

Algorithm

Count the occurrences of each string by filling up a hashtable of string -> count.
Iterate through the hashtable, putting each [count, string] pair into a min-heap (sorted by count) of size k.
O(n + u * log(k)), n = number of strings, u = number of unique strings

Implementation

Located at: http://github.com/vadimg/luajit-talk

Implemented in:

C++
Javascript (v8)
Lua using C API
Lua using LuaJIT FFI

Data is a list of 1 million strings, 1 per line, in a file.

Common interface to data for all implementations (data.h)

Benchmark tests: C boundary, hashtable, array, function call, string operation speed

LuaJIT Limitations

On x64, can't allocate more than 2GB of Lua objects
On x64, all Lua memory allocations must be in the first 2GB region of the address space (http://hacksoflife.blogspot.com/2012/12/integrating-luajit-with-x-plane-64-bit.html)
On x64, can't provide your own allocation functions
Garbage collector is slow (same as standard Lua)

Takeaway

LuaJIT is really, really fast
The FFI is really, really fast: you don't have to worry much about the boundary between C and Lua
You can get all the flexibility and development speed of a dynamic language, with the performance of C
LuaJIT's FFI is so easy to use that exposing C functions to Lua is no longer a pain
Consider using it in your C/C++ projects :)