StateFun.jl - Flink Stateful Functions Julia SDK

Setting up Julia

• download Julia 1.6 … i put in /opt
• expose binary: sudo ln -s /opt/julia-1.6.2/bin/julia /usr/local/bin/julia-1.6
• start julia repl: julia-1.6

VsCode

• Install vscode
• Add the Julia language extension in vscode settings
• Set the executable to /usr/local/bin/julia-1.6
• Start a julia repl in the Command Palette:
  • Julia - Restart Language Server (might need a restart depending on the last state of your IDE)
  • Julia - Start REPL

Create a new package

I’ll follow along with the instructions here: https://syl1.gitbook.io/julia-language-a-concise-tutorial/language-core/11-developing-julia-packages

• In the julia repl, press ] to enter package management.
• Press ? to get help at any point.
• generate StateFun will generate some stub files for our package.
• dev StateFun registers the new package (located in /opt/StateFun for me) in the Julia environment
• Go to github, create a new repo: StateFun.jl
• Set up the repo and push:

git init
git add -A
git remote add origin git@github.com:tbreloff/StateFun.jl.git
git checkout -b main
git commit -am "initial commit"
git push -u origin main

Then you can tell Julia about your github address:

(@v1.6) pkg> rm StateFun
(@v1.6) pkg> add git@github.com:tbreloff/StateFun.jl.git
(@v1.6) pkg> dev StateFun

Setup:

(@v1.6) pkg> activate StateFun
(StateFun) pkg> add Revise Test
(StateFun) pkg> test StateFun
...
     Testing Running tests...
     Testing StateFun tests passed

Building StateFun proto files

In Julia REPL:

]activate; add ProtoBuf  # note: you must have ProtoBuf installed in the base environment for this to work.
]activate StateFun
include("/opt/StateFun/scripts/download_statefun_protos.jl)
include("/opt/StateFun/scripts/generate_protos.jl)

Workflow

We’re going to borrow the python version of the greeter example in flink-statefun-playground, except we’ll swap out the python service for a julia service.

Python version

First get the python version working:

• clone statefun examples: git clone https://github.com/apache/flink-statefun-playground.git && cd flink-statefun-playground
• checkout release branch: git checkout release-3.0
• cd python/greeter
• install docker compose and then docker-compose build && docker-compose up -d
• Examine the logs: docker-compose logs -f functions

Julia Statefun Functions

The Julia server

We’ll make a very simple http server using Mux.jl, and save it as scripts/run_server.jl:

using Mux, StateFun, ProtoBuf

@app app = (Mux.defaults, # use Mux.prod_defaults later
    page("/statefun", req -> begin
        # ...
    end),
    Mux.notfound()
)

# ...
wait(serve(app, port))

This server will listen at http://localhost:8000/statefun and handle anything that comes in. This is the same address that the python greeter example was listening.

Wrap our Julia server in a Docker image

This is the Dockerfile which will run our Julia server:

FROM julia:1.6.2

WORKDIR /opt/StateFun

# copy this package into the /opt directory in the container
# ensure this is a git repo... Pkg needs it when adding
RUN apt-get -y update && apt-get -y install git
RUN git init

# warm up
COPY Project.toml .
COPY Manifest.toml .
RUN julia -e 'using Pkg; Pkg.activate("."); Pkg.precompile()'

# add the package and precompile it
COPY src src
COPY scripts/run_server.jl .

# ensure the package's environment will be activated on julia startup
RUN mkdir -p /root/.julia/config
RUN echo 'using Pkg; Pkg.activate(".")' > /root/.julia/config/startup.jl

# run julia
ENTRYPOINT ["julia", "run_server.jl"]

You should then build it:

docker build -t statefunjl/greeter_example .

Swap out python for julia service

In the docker-compose.yml file, replace the functions block with this:

  functions:
    image: statefunjl/greeter_example
    expose:
      - "8000"

This is going to launch the julia docker container we made instead of a python version.

Run it all

First:

cd /path/to/flink-statefun-playground/python/greeter

Now we’re going to start up the docker-compose cluster which will contain:

• A Flink cluster (with Statefun Functions enabled)
• A kafka/zookeeper deployment
• A fake-data producer
• Our Julia service to handle Statefun Functions events (with docker-compose name functions)

docker-compose up -d

After a few seconds, the containers should be running and your Julia handler should be logging what it sees:

docker-compose logs -f functions