Watching docker logs in the browser with Python and Eel

Share

Multiple microservices deployed with Kubernets and Docker mean multiple containers to check for logs when debugging or watching internal communication. It would be handy if we could look through logs in a more organized way - and that's how Docker-watcher was born.

Application idea and OKRs in IT

Working with Social WiFi and ChatPilot codebases means I have to work with multiple celery workers, socket.io servers and flask API microservices. It's quite common to check logs to see what a worker or server is doing. To make it more streamlined and easier to watch multiple containers at once I've make a plan to make a simple tool that would help with this.

This tool was planned as one of key results of an Improve quality of life while coding OKR at Social WiFi. Objectives Key Results aren't common in IT but we decided to try OKRs in our company. It's not something that was a part of a sprint or planned to do by the company directly.

The idea was to use and existing tool or framework/library to do some sort of a grid view of shell terminals displaying logs from selected docker containers. With a group by microservices drop down menu at the top:

Something simple that won't take a lot of time to develop and that is not complex to build/deploy.

I've started looking at options - either shell terminal widgets/apps or some Python solutions. I've checked tkterminal which is a shell widget for tkinter but it seems to have some limitations when running continuous/locking commands. qtermwidget was bit to problematic to build on Xubuntu for PyQt so I've just reverted to my core technology stack and started looking at some semi-web based platforms. That's when I found Eel and after a quick proof of concept with container logs stream I've decided to use it.

Python eel

eel is an electron-alike tool to make simple applications in Python that have a HTML/CSS/JS interface. It uses websockets and gevent to make it happen. As you can see on the github repo the project currently lacks an active maintainer which led to multiple forks, pull requests and issues. It's not the best pick long term but it got the job done for docker-watcher.

As it uses gevent monkey patching and custom logic over websockets it's kind of a black box. If it doesn't work you may not know why and what to do. It's not the best pick for production and mission critical applications. If it would be actively maintained with even bigger user base then maybe it would look better.

Python calling JavaScript functions and JavaScript calling Python functions?

In your Python code you can call a JavaScript function with a simple eel.my_javascript_function(arguments, ...). What this does is sends via websockets an event to Eel JavaScript code. It receives the function name and arguments and then calls said function with given arguments. It's not literal direct call, rather than a bit of serialization and websockets communication.

AJAX request can send an asynchronous request from the browser to the backend but backend can't do that. To alleviate this a websocket standard was developed. It allows server-client communication via events. JavaScript code working in the browser can connect to the server and wait for events. Backend websocket server can then send them at any moment. A sort of reversed AJAX.

As websocket implementation is browser specific there are wrappers that handle all the differences like socket.io. I recommend this over raw websocket usage.

To call frontend functions from backend you have to expose them in JavaScript via eel.expose(FUNCTION_NAME);. For backend code to be called by JavaScript you have to you have to use eel decorator: @eel.expose.

Docker Watcher

The application is a simple usage of Docker API Python library, some logs/container name grouping and eel. The code is structured using use cases:

• docker_cases: there are three use cases here - get containers for microservices from settings, stream logs for a container and get current container ID for given container name.
• microservices_cases: This uses previous use case and microservices list from settings to group containers by microservice.
• eel_cases: two use cases used directly in the eel app - they call existing use cases and serialize the results.

As you could notice those use cases return dataclasses which are used internally between use cases and serialize to JSON safe data types before using them in eel exposed functions as arguments.

Using dataclasses defines a clear interface of the object that given function or method returns. If it would be a dictionary you would have to look at the code to see what's the structure of it or have an excess comments in the code documenting the code.

Use cases is one of ways of structuring code where business logic is extracted to high level functions/classes with very explicit business naming that then go lower and perform needed logic. That way each layer is separated. Raw docker operations on containers and logs, then microservices, then eel serialized layer and then eel itself. It's easier to test, reuse and refactor such code. You could take docker and microservice use cases and use it with some other presentation layer quite easily.

Web interface

Eel uses a browser to make the UI happen so you can use all of the HTML/CSS/JS capabilities. The small trick is that eel backend will call frontend JavaScript functions so some parts of the interface have to be build with JavaScript. SPA JS frameworks like Ember, Vue would usually be used for anything more than basic page, but here I wanted to keep it simple so I've used the good old jQuery and Bootstrap for basic typography and styles.

The UI structure is quite simple, but there is one trick here:

<!DOCTYPE html>
<html lang="en">
  <head>
    <title>Docker Watcher</title>
  </head>
  <body>
    <div id="application">
      <nav class="navbar navbar-expand-lg navbar-dark bg-dark">
        <div class="container-menu">
          <div class="menu-elements"></div>
        </div>
      </nav>
      <div class="logs"></div>
    </div>
    <div id="widgets">
        HIDDEN WIDGETS HERE
    </div>
  </body>
</html>

Everything visible is in the application DIV. The widgets DIV is hidden via CSS. As jQuery (and not a nice SPA framework) is creating the UI dynamically based on incoming data (list of microservices, list of log entries etc.) I have to keep a HTML template to be used for such item.

For example Bootstrap drop down menu is adapted for a microservice drop-down menu template:

<div class="collapse navbar-collapse">
    <ul class="navbar-nav">
    <li class="nav-item dropdown">
        <a class="nav-link dropdown-toggle" href="#" role="button" data-bs-toggle="dropdown" aria-expanded="false">
        [service name]
        </a>
        <ul class="dropdown-menu dropdown-menu-dark">
        [dropdown items]
        </ul>
    </li>
    </ul>
</div>

jQuery then takes such piece of HTML, clones it, fills with data and inserts into a specific container in the visible part of the app. Check buildMicroservicesMenu to see how it's implemented. For each microservice it receives from eel backend it clones the widget, sets the name and then fills the drop down elements based on containers found for given microservice. The jQuery code is bit crude though, it still could have been done bit cleaner.

addLogWidget is called on a click from the microservice drop down menu - it checks if log widget for given container is present and if not it clones and inserts the log box template and calls a eel backend function "start_log_stream" that starts streaming logs for container given by name.

appendLog JS function would be called by the log streaming process on the backend. It has a similar flow to the previous functions - finds a log widget for given container and then inserts it into the DIV containing all the logs and scrolls it down so that so you always see the latest logs.

Asynchronous backend

Eel uses Bottle and Gevent. It does not spawn full POSIX threads but it does support asynchronous operations via green threads and context switching. Context can switch between main and side loops as needed:

import eel


def get_logs():
    while True:
        print('get_logs')
        eel.sleep(1)


eel.init('web')
eel.spawn(get_logs)
eel.start('index.html', block=False)

while True:
    print("I'm a main loop")
    eel.sleep(1)

Here where eel.sleep comes in. eel.spawn spawns a greenlet (Gevent green thread) that will call the specified function. If it's blocking like in this case it will continuously print get_logs while Python thread will be stuck in the I'm a main loop loop. eel.sleep allows switching context between greenlets and main thread.

In this example you would see one main and one get_logs entry repeated over and over. If the get_logs function would have a lot of data to send/process at some point you would see that I'm a main loop would not show up for a while as the context will be moved and locked temporarily on the get_logs.

Docker Watcher spawns a greenlet for each log streaming container. As the log flow isn't that high on the development platform the context switching of eel will be sufficient to show few container logs pretty much live side by side.

Comment article