In this hands-on session you will setup an environment to locally develop, debug, and execute Flink programs. Furthermore, you will install a Flink instance to locally executed Flink programs on your machine.

1. Software requirements

Install VirtualBox and import the following VM image: http://amir.sics.se/public/tools/images/flink.ova

Checkout the Flink repository and import it into IntelliJ:

git clone https://github.com/dataArtisans/eit-summerschool-15-exercises.git

In IntelliJ press: File => New => Project from Existing Sources.

When asked, provide the path to the cloned repository and select Maven as project type.

Flink programs can be executed and debugged from within an IDE. This significantly eases the development process and gives a programming experience similar to working on a regular Java application. Starting a Flink program in your IDE is as easy as starting its main() method. Under the hood, the ExecutionEnvironment will start a local Flink instance within the execution process. Hence it is also possible to put brakepoints everywhere in your code and debug it.

Assuming you have an IDE with a Flink project imported, you can execute the example WordCountExample program which is included in the project as follows:

  • Open the WordCountExample class in your IDE
  • Press the Run button in the top toolbar, which executes the main method of the class.

In order to execute program on a running Flink instance (and not from within your IDE) you need to install Flink on your machine. To do so, follow these steps:

  • Download a Flink 0.9.0 build of Apache Flink. The latest build can be downloaded here.
  • Extract the downloaded .tgz archive
  • The resulting folder contains a Flink setup that can be locally executed without any further configuration.

Given that you have a local Flink installation, you can start a Flink instance that runs a master and a worker process on your local machine in a single JVM. This execution mode is useful for local testing.

On UNIX system you can start a Flink instance as follows:

cd /to/your/flink/installation
./bin/start-local.sh

You can validate that a local Flink instance is running by looking at the log files in ./log/ or opening the JobManager’s webinterface at http://localhost:8081.

./bin/stop-local.sh