JENNIFER Python

Python language, which has been receiving attention lately in the data processing sector, naturally expended the development experience to the web by the developer, and in addition to this, services using web frameworks such as Django and Flask started to settle down as part of the micro service even in the enterprise environment.

In accordance with such changes, JenniferSoft released the first version supporting monitoring for the Python environment. This document explains the unique features and operating methods of the JENNIFER Python agent.

Supporting Environment

It is well-known that Python language is platform-independent. But from the perspective of APM products, it is featured by more restrictions compared to the Java/.NET, etc. environments because of the active development of the Python language specifications and its surrounding run-time environment.

Supports only Linux

Strictly speaking, frameworks such as Django/Flask can be executed irrelevant to the platform. But most service environments are hosted through exclusive web applications that support WSGI interfaces such as uwsgi or gunicorn, and the problem here is that this application program only supports Linux.

The fundamental reason for this is because of Python interpreter’s full lock known as “GIL (Global Interpreter Lock.” A single thread that acquired GIL can process the byte code of Python. This is not a big problem in normal environments, but in the case of web application environments that process multiple requests based on threads, severe performance drop issues occur due to GIL.

In order to bypass this, the WSGI web application (uwsgi, gunicorn, etc.) selected the method of dividing the process with a fork instead of using threads for Python. Therefore, Django/Flask can be executed in the Windows environment as well, but uwsgi/gunicorn that hosts it cannot support Windows (due to the use of fork), and thus, JENNIFER Python currently only supports monitoring of the Linux environment.

Supporting specific Python run time and packages

Python language does not have good sub-compatibility for version upgrades. The JENNIFER Python module uploaded within the process also uses Python language internally, and therefore, it is affected by sub-compatibility because it has the same run-time restrictions.

This problem is applied not only in the Python run-time but also for the package as well. As most packages do not have sub-compatibility and because the features of APM products require close interworking with the internal codes of the target package, it has the problem of having to add support for each package version.

In result, the Python web application operated by the client website must meet the run-time and package version conditions supported by JENNIFER Python products to monitor at desired levels. If not, it requires additional time for adding functions to JENNIFER Python through prior negotiations for packages not yet supported or its ensuing version.

Operating Structure

As explained above, uwsgi or gunicorn, etc. that host the Python web application program has operating methods based on forks by default. It therefore has to adopt a structure different from existing Java/.NET agents that are activated in single processes.

In other words, the JENNIFER module must be loaded for operation for each forked process, but each process cannot be identified as a single instance in JENNIFER View. Thus, it requires a separate process that represents the single application program, while communicating with the fork process. JENNIFER Python reflects this to be activated through the following structure.

The proxy process communicates through the JENNIFER data server and single TCP connection, and it communicates with the internal fork process via UNIX Socket.

Thanks to this structure, the monitoring screen shown in the JENNIFER Console is not too different from Java or .NET.

Installation and Uninstallation

The Python ecosystem has a well-organized package archive called PyPI. JENNIFER Python is also distributed here and currently, it can be read in the following address.

jennifer-python: https://pypi.org/project/jennifer-python/

Therefore, the installation uses pip just like other Python packages.

$ pip install jennifer-python

The first thing to do after installation is to create an INI configuration file containing connection information to the “JENNIFER data server.” This file can be written manually as well, but the basic INI file can be created easily with the following using the “jennifer-admin” command installed together with the “jennifer-python” package.

$ jennifer-admin generate-config

Then the “jennifer.ini” file is generated with the following contents based on the directory from which the command was executed.

[JENNIFER]

server_address = 127.0.0.1
server_port = 5000
domain_id = 1000
inst_id = -1
log_path = /tmp/jennifer-python-agent.log

Afterward, the values for each field can be set according to the following criteria using the editor.

The jennifer.ini file must be created for every Python application file, and just the inst_id value can be set differently.

The Python application program subject to monitoring through the corresponding INI file configuration must be executed by connecting to the jennifer-python module. At this time, by using the method of setting the INI file path together with environmental variables, the Python web application program can be executed in the following format.

$ JENNIFER_CONFIG_FILE=<Configuration file path> jennifer-admin run <Existing python execution code>

If your application program was executed with the existing “uwsgi -i uwsgi.ini,” and the created jennifer.ini file path is /home/user/jennifer.ini, then it can be executed as follows.

$ JENNIFER_CONFIG_FILE=/home/user/jennifer.ini jennifer-admin run uwsgi -i uwsgi.ini

If all settings are correct, the following log message will appear on the uwsgi execution screen.

…[Omitted]…
---------------- [App Initialized] ----------------------
MachineName =  TESTPC
Is64BitProcess =  ('64bit', 'ELF')
Python Version =  3.8.10
Jennifer Python Agent Install Path =  /usr/local/lib/python3.8/site-packages/jennifer
Jennifer Python Agent Domain ID =  2790
Jennifer Python Agent Inst ID =  601
Jennifer Python Agent Pid =  4925
---------------------------------------------------------
…[Omitted]…

Meanwhile, to download the agent from the web application program, simply execute using the existing method without going through “jennifer-admin.” In addition, to delete the JENNIFER Python binary, simply remove it using pip.

$ pip uninstall jennifer-python

Support Function

Monitoring shown in the JENNIFER Console screen satisfies Active Service view and X-View profile, thus allowing monitoring a similar experience as existing JENNIFER Java/.NET at basic levels.

The profile subject currently supports HTTP and DB summoning as follows at the time for which this was written (5/20/2022).

• urllib, requests
• mysql, redis, mongo db, sqlite3

This is still in its early version and many functions are not yet available in the existing JENNIFER Java/.NET product, so please be patient as we make many improvements continuously.