We’ve been testing some Spark code that will eventually be moved to AWS. For now, to save costs, we’ve created a 8 node Spark cluster that runs on a set of Virtual Machines running Ubuntu on VirtualBox. We’ve developed some bash-scripts to make starting (and shutting down) the VMs easy.
In the shutdown part, we were using the ‘poweroff’ option to vboxmanage, which is the equivalent of yanking the power supply out, and the HD wont shutdown properly. Yikes!
The correct way to shut them down is:
VBoxManage controlvm Spark_Worker_1 acpipowerbutton