In this hands-on session you will setup an environment to locally develop, debug, and execute Flink programs. Furthermore, you will install a Flink instance to locally executed Flink programs on your machine.
1. Software requirements
Install VirtualBox and import the following VM image: http://amir.sics.se/public/tools/images/flink.ova
2. Import Flink Maven project
Checkout the Flink repository and import it into IntelliJ:
git clone https://github.com/dataArtisans/eit-summerschool-15-exercises.git
In IntelliJ press:
File => New => Project from Existing Sources.
When asked, provide the path to the cloned repository and select Maven as project type.
3. Execute and debug a Flink program in an IDE
Flink programs can be executed and debugged from within an IDE. This significantly eases the development process and gives a programming experience similar to working on a regular Java application. Starting a Flink program in your IDE is as easy as starting its
main() method. Under the hood, the
ExecutionEnvironment will start a local Flink instance within the execution process. Hence it is also possible to put brakepoints everywhere in your code and debug it.
Assuming you have an IDE with a Flink project imported, you can execute the example
WordCountExample program which is included in the project as follows:
- Open the
WordCountExampleclass in your IDE
- Press the Run button in the top toolbar, which executes the main method of the class.
4. Install Flink for local execution
In order to execute program on a running Flink instance (and not from within your IDE) you need to install Flink on your machine. To do so, follow these steps:
- Download a Flink 0.9.0 build of Apache Flink. The latest build can be downloaded here.
- Extract the downloaded
- The resulting folder contains a Flink setup that can be locally executed without any further configuration.
5. Start a local Flink instance
Given that you have a local Flink installation, you can start a Flink instance that runs a master and a worker process on your local machine in a single JVM. This execution mode is useful for local testing.
Start local Flink instance
On UNIX system you can start a Flink instance as follows:
cd /to/your/flink/installation ./bin/start-local.sh
Validate that Flink is running
You can validate that a local Flink instance is running by looking at the log files in
./log/ or opening the JobManager’s webinterface at http://localhost:8081.
Stop local Flink instance