Lima-VM: docker-desktop alternative for MacOSX

On August 31st, Docker surprises the world with a news about the docker-desktop application: it won't be free anymore. Even if it is a normal and legit decision, this can be a hard decision for big companies as the final invoice can have consequences on the IT budget.

The same day I discovered, thanks to AkihiroSuda Medium Post, that a possible alternative for MacOSX exists... but I needed to make some enhancements to what was described in the post as my specific use cases required more things.

Installation

The lima installation, thanks to homebrew, is really simple and everything is managed:

brew install lima

Once finished there are 2 main commands available on your PC: lima to access to the virtual machine and execute "linux" commands; limactrl to control the machine, create, start, stop, ...

As Lima is automatically forwards all the VM ports to the host and shares the volumes, everything is as easy as with the docker-desktop. The main difference is that by default it is not using the docker-engine but the containerd directly instead. But for a standard / simple usage this is enough.

Run and Use containers

The CLI to use to interact with the default containerd is nerdctl... but in the end the big difference is only in the script name, because all the commands (even the compose one) are there.

To simplify the usage, just add an alian on your Mac to directly execute the right command:

alias docker="lima nerdctl"

Docker API

Containerd is not exposing an API like the docker one, this means for some application it is impossible to interact and control docker. For example, in Java unit test with Testcontainers this interaction is mandatory. But Lima is Linux and Docker is OSS... so you can configure it to use docker instead of containerd. As AkihiroSuda suggested in its post comments, this is quite simple:

curl -fsSL https://get.docker.com | lima
lima dockerd-rootless-setuptool.sh install

Then you have to access to the docker.sock from your Mac... this can be done with the following command:

ssh -p 60022 -i ~/.lima/_config/user -o NoHostAuthenticationForLocalhost=yes -L ~/docker.sock:/run/user/$(id -u)/docker.sock 127.0.0.1

Once done you will have a docker.sock file in your home folder. Just configure the application requiring the Docker API, to use the socked within your home folder. To test if everything is working well, from your Mac you can run the following command that is equivalent to run the docker images command.

curl --unix-socket ~/docker.sock http://localhost/images/json

image.png

It is working but is quite annoying in the end, anytime you need to interact with the Docker API you have to remember to run this command.

WARNING: before to run the ssh command, check if the docker.sock file already exists on your Mac. If it was not deleted by a previous execution, ssh couldn't create a new one with the same name. So nothing will work in this case.

image.png

Private Registry

If you need to use a private registry, you need to be sure to have anything required to login installed inside the VM. In my case, I'm always using GCR/GAR Registry... so I needed to install the gcloud package inside the lima VM.

echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
sudo apt-get install apt-transport-https ca-certificates gnupg
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
sudo apt-get update && sudo apt-get install -y google-cloud-sdk

Then just connect the docker to the desired gcloud registry:

export DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock
gcloud auth configure-docker --quiet
docker login xxxx

NOTE: you surely need to login to gcloud to be able to use the private docker registry (gcloud auth login).

Package everything in a configuration file

Instead of configuring all the things manually everytime, you can benefit from the lima.yaml file and package everything you need. The important part is that all the command must be idempotent as they are executed anytime you restart the VM. Here an example of my script:

provision:
  # `system` is executed with the root privilege
  - mode: system
    script: |
      #!/bin/bash
      set -eux -o pipefail
      if ! apt list --installed | grep docker-ce; then
        curl -fsSL https://get.docker.com | sh -
        echo 'export DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock' > /etc/profile.d/docker.sh
      else
        echo "Docker already installed"
      fi

      if ! apt list --installed | grep google-cloud-sdk; then
        echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
        apt-get install apt-transport-https ca-certificates gnupg
        curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
        apt-get update && apt-get install -y google-cloud-sdk
      else
        echo "Google Cloud already installed"
      fi


  # `user` is executed without the root privilege
  - mode: user
    script: |
      #!/bin/bash
      set -eux -o pipefail
      dockerd-rootless-setuptool.sh install
      gcloud auth configure-docker --quiet

WARNING: The first creation takes long time (several minutes depending on the Mac performances). When the lima creation seems finished, it is not. You can follow all the creation logs with a tail -f ~/.lima/default/serial.log and everything is finised when you can read

[  OK  ] Finished Execute cloud user/final scripts.
[  OK  ] Reached target Cloud-init target.

Now you can execute all the docker commands using:

lima docker xxxx

So you can just add the correct alias on your Mac machine:

alias docker="lima docker"

WARNING: take care because containerd and nerdctl are also installed on the machine, but they are not sharing things with the docker engine. This means if you run a container in containerd, the downloaded image cannot be seen and used by the docker-engine: the first time you start the same machine in docker, it must download it.

image.png

image.png

Create the full Lima VM

If you want to provision a Lima VM like the one I described, you can use my sample file.

Executing it in the following way:

limactl start default.yaml

EDIT: 07/09/2021 Script is now updated with some interesting enhancements:

  • probes are added to wait until the full installation completion. It is done with 3 added steps: docker-ce, gcloud and user configuration

image.png

  • The rootless docker configuration is now exposing the docker API over TCP too
- mode: user
    script: |
      #!/bin/bash
      set -eux -o pipefail
      dockerd-rootless-setuptool.sh install
      if ! grep DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS ~/.config/systemd/user/docker.service; then
        /usr/bin/sed -i '/Environment=.*/a Environment=DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS="-p 0.0.0.0:2375:2375/tcp"' ~/.config/systemd/user/docker.service
        /usr/bin/sed -i 's/ExecStart=.*/ExecStart=\/usr\/bin\/dockerd-rootless.sh -H tcp:\/\/0.0.0.0:2375/g' ~/.config/systemd/user/docker.service
      else 
       echo "Docker service already configured"
      fi
      /usr/bin/systemctl --user daemon-reload
      /usr/bin/systemctl --user restart docker.service
      gcloud auth configure-docker --quiet
  • The API port (2375) is automatically exposed to the host server. You can now use any docker depend app just executing export DOCKER_HOST=tcp://localhost:2375
    portForwards:
    - guestPort: 2375
      hostIP: "127.0.0.1"
    
    There is no need to create an SSH port forward with the docker.sock anymore (but it will be available too as the rootless docker is exposing the API with both two methods).

EDIT: 08/09/2021 The latest version of the gist is tested against several project with testcontainer with success results. On the host machine we just need to configure the testcontainers/ryuk to use the correct socket.

 export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/run/user/502/docker.sock

The UID (502) may differ in your VM. To get the correct one you can run

lima echo "/run/user/$(id -u)/docker.sock"

It won't ever change if you won't create your VM again and you can add in your host .zshrc or .bashrc or whatever rc file.

It should provide a better docker and provision experience.