Jupyterlab not working on clean Ubuntu 20.04 LTE

Just installed a clean Ubuntu 20.04 LTE with heavyai 6.1.1 docker gpu free edition, and jupyterlab.

The problem is Jupyterlab container is always in “exited” status (it seems not normal to me) , and there isn’t any Jupiter button in Immerse or Heavysql Editor.

Docker and nvidia cuda are working fine, here some detail of this installation:

nvidia cuda:

/var/lib/heavyai$ docker run --rm nvidia/cudagl:11.0-runtime-ubuntu18.04 nvidia-smi
Mon Aug 29 11:47:13 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.65.01    Driver Version: 515.65.01    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 28%   58C    P2    43W / 180W |   1163MiB /  8192MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

docker container with exited jupyterlab:

docker ps -a
CONTAINER ID   IMAGE                                 COMMAND                  CREATED      STATUS                         PORTS                                                                                                      NAMES
7006f54fde0f   omnisci/omnisci-jupyterhub:v0.4       "/docker-entrypoint.…"   3 days ago   Up About an hour               8000/tcp                                                                                                   jupyterhub
8b9874edefdd   heavyai/heavyai-ee-cuda:latest        "/bin/sh -c '/opt/he…"   4 days ago   Up About an hour               0.0.0.0:6273-6274->6273-6274/tcp, :::6273-6274->6273-6274/tcp, 0.0.0.0:6278->6278/tcp, :::6278->6278/tcp   heavyaiserver
aabd5c89e436   omnisci/omnisci-jupyterlab-cpu:v0.5   "tini -g -- echo"        4 days ago   Exited (0) About an hour ago                                                                                                              heavyai_jupyterlab-tmp_1

docker-compose.yml.txt (5.1 KB)
heavyai.conf.txt (92 Bytes)
servers.json.txt (38 Bytes)

User and role created and assigned:

heavysql> \roles
heavyai_jupyter
heavysql> \role_list user1
heavyai_jupyter

folders tree:

WS:/var/lib$ 
gc@gc-Z10PE-D16-WS:/var/lib$ ls -la |grep heavyai
drwxr-xr-x  5 gc      root          4096 ago 29 14:04 heavyai
gc@gc-Z10PE-D16-WS:/var/lib$ cd heavyai
gc@gc-Z10PE-D16-WS:/var/lib/heavyai$ ls -la
total 36
drwxr-xr-x  5 gc root     4096 ago 29 14:04 .
drwxr-xr-x 75 root     root     4096 ago 24 11:30 ..
drwxr-xr-x  3 root     root     4096 ago 25 12:33 data
-rw-rw-r--  1 gc gc 5265 ago 29 12:34 docker-compose.yml
-rw-rw-r--  1 gc gc   92 ago 29 12:27 heavyai.conf
drwxr-xr-x  2 root     root     4096 ago 24 13:20 jupyter
-rw-rw-r--  1 gc gc   38 ago 25 15:44 servers.json
drwxr-xr-x 10 root     root     4096 ago 24 13:08 storage
gc@gc-Z10PE-D16-WS:/var/lib/heavyai$ cd jupyter
gc@gc-Z10PE-D16-WS:/var/lib/heavyai/jupyter$ ls -la
total 8
drwxr-xr-x 2 root     root 4096 ago 24 13:20 .
drwxr-xr-x 5 gc root 4096 ago 29 14:04 ..
gc@gc-Z10PE-D16-WS:/var/lib/heavyai/jupyter$ 

The Jupyterlab container log is empty:

WS:/var/lib/heavyai/jupyter$ docker logs --tail 50 --follow --timestamps aabd5c89e436
2022-08-25T09:56:07.857247368Z 
2022-08-25T10:03:21.664031064Z 
2022-08-25T10:06:12.291297711Z 
2022-08-25T10:36:50.836189185Z 
2022-08-25T10:40:12.694607414Z 
2022-08-25T10:49:16.171701240Z 
2022-08-25T10:50:03.308175355Z 
2022-08-25T11:07:19.288349578Z 
2022-08-25T11:29:24.304789204Z 
2022-08-25T11:44:55.033777252Z 
2022-08-25T13:13:37.071985704Z 
2022-08-25T13:25:59.243723916Z 
2022-08-25T13:39:00.748249237Z 
2022-08-29T10:34:57.008028066Z 
gcampana@gcampana-Z10PE-D16-WS:/var/lib/heavyai/jupyter$ 

This is a clean Ubuntu 20.04 image and straightforward Docker/heavyai/Jupyter installation, what should I still check ?

Any help ?

Hi Gianfranco,

I’ve made some revisions to your docker-compose file and attached them here. It looks like you’re not overriding the default start command, and thus the default start command would expect for a configuration file to be at /var/lib/heavyai/heavy.conf not /heavyai-storage/heavyai.conf, and further for your data folder within the container to exist at /var/lib/heavyai/.

Additionally, the environmental variables used by the jupyterhub/jupyterlab containers haven’t been updated, so I’ve ammended these as well.

On our end, it looks like our jupyter install page needs an update; sorry for the inconvenience.

heavy.conf.txt (92 Bytes)
docker-compose.yml.revised.txt (5.1 KB)

Hope this helps! Look forward to your feedback.

Thanks,
Neill

Thanks for your help Neil,
using your files improved the situation.

Now I can see the Jupyter button with the “admin” user (not valid for jupyerlab access because I can’t assign roles to admin user), but I don’t see the Jupyterlab button with the “user1” user, to which I assigned the heavyai_jupyter role and all privileges on database:

heavysql> grant all on database heavyai to user1;
heavysql> \role_list user1
heavyai_jupyter

In addition, often the http sessions don’t disconnect and are left active: here I have two http sessions active, but I don’t know how to re-enter properly in the browser in order to log-out , or how to kill (stop) the session in order to free the session.

heavysql> show user sessions ;
session_id|login_name|client_address|db_name
321-9azc|admin|http:localhost:48732|heavyai
428-R1kc|admin|tcp:localhost:56668|heavyai
838-kdaT|user1|http:172.19.0.1|heavyai

If I try to re-enter the session in browser with both “admin” and “user1” login, I get:
image.

and I imagine hit this error could be the cause the session is left active and pending.
What does “JSON.parse: unexpected end of data at line 1 column 1 of the JSON data” exactly mean, and what should I do to avoid that error?

I added a ten minutes timeout in heavyai.conf but it would be nice to manage (and kill) sessions if needed: from what moment will this parameter (and in general parameters edited in heavyai.conf) be active?

idle-session-duration 10

Update: after stopping and restarting heavyai container the Jupyter button is missing with both users admin and user1: it seems that the “JSON.parse:” error is gone, now.

Hi Gianfranco,

Now I can see the Jupyter button with the “admin” user (not valid for jupyerlab access because I can’t assign roles to admin user), but I don’t see the Jupyterlab button with the “user1” user, to which I assigned the heavyai_jupyter role and all privileges on database:

Can you please try to use the role “omnisci_jupyter” instead and also make this change in docker-compose.yml?

Additionally, under optional settings, try to add this line to allow your admin user access:
OMNISCI_ALLOW_SUPERUSER_ROLE: "true" (note rename from HEAVYAI_ to OMNISCI_ if uncommenting existing line).

Note that a full docker compose up -d is required for changes in docker-compose.yml to be enabled.

From what moment will this parameter (and in general parameters edited in heavyai.conf) be active?

These settings require a restart of heavydb and/or heavy_web_server.

HI Neil,
I followed your hints but I still have no Jupyterlab button in Immerse with both the user “Admin” and “user1”.
Role omnisci_jupyter is created and granted, and OMNISCI_ALLOW_SUPERUSER_ROLE: “true” is set and activated with compose up -d and restart of heavydb and heavy_web_server.

heavysql> \role_list user1
heavyai_jupyter
omnisci_jupyter

docker-compose.yml.txt (5.2 KB)
heavyai.conf.txt (117 Bytes)
servers.json.txt (38 Bytes)

This is the heavyai cotainer log, I see something strange:

2022-09-01T07:05:46.529369127Z - heavydb 17 started
2022-09-01T07:05:46.529950585Z - heavy_web_server 18 started
2022-09-01T07:05:49.382755191Z ⇨ http server started on [::]:6273
2022-09-01T07:05:51.532355298Z Navigate to: http://localhost:6273
2022-09-01T07:13:30.556937397Z Failed to write to log, write /var/lib/heavyai/storage/log/heavy_web_server.37460dcae923.root.log.ALL.20220901-070546.18: file already closed
2022-09-01T07:13:30.666992326Z 2022-09-01T07:13:30.666678 E 17 0 7 HeavyDB.cpp:194 Exception: Thrift server exited: Cannot assign requested address: port 6278
2022-09-01T07:13:30.667022606Z 2022-09-01T07:13:30.666762 E 17 0 9 HeavyDB.cpp:194 Exception: Thrift server exited: Cannot assign requested address: port 6276
2022-09-01T07:13:30.667093997Z 2022-09-01T07:13:30.666719 E 17 0 8 HeavyDB.cpp:194 Exception: Thrift server exited: Cannot assign requested address: port 6274
2022-09-01T07:13:32.561755610Z startheavy 7 exited
2022-09-01T07:13:34.301915394Z startheavy 7 running
2022-09-01T07:13:34.312219693Z WARN: config file does not exist, ignoring: --config /var/lib/heavyai/heavy.conf
2022-09-01T07:13:34.312290332Z Backend TCP:  localhost:6274
2022-09-01T07:13:34.312313508Z Backend HTTP: localhost:6278
2022-09-01T07:13:34.312319797Z Frontend Web: localhost:6273
2022-09-01T07:13:34.312325600Z Calcite TCP:  localhost:6279

It is searching /var/lib/heavyai/heavy.conf, this seems an error

Just an update, heavyai.conf is found correctly now.

The problem of the missing Jupyter button still persists.

I observer in dockerfile.cuda the startheavy command calling heavy.conf insetad of heavyai.conf


CMD /opt/heavyai/startheavy --non-interactive --data /var/lib/heavyai/storage --config /var/lib/heavyai/heavy.conf

and also in the container log the entry:

2022-09-01T12:32:50.948375791Z WARN: config file does not exist, ignoring: --config /var/lib/heavyai/heavy.conf

so I copied my heavyai.conf as heavy.conf and the Jupyter button magically appeared :slight_smile:

Now, (hopefully) I need a last step in order to complete the installation: opening Jupyter from the user “user1” leads to :

but actually the omnisci_jupyter_role is granted to that user:

heavysql> \role_list user1
heavyai_jupyter
omnisci_jupyter
heavysql> 

Any help for the role issue?

Well, i renamed the parameter name in the yml file and now the role is found on the user.
But a different error is returned in Jupyterlab :

The user “user1” is already connected to Heavyai and Immerse, therefore it isn’t an authentication problem.

I still have this error in the container log, this could be the cause:

2022-09-02T14:07:47.911421271Z 2022-09-02T14:07:47.911066 E 17 0 9 HeavyDB.cpp:194 Exception: Thrift server exited: Cannot assign requested address: port 6278
2022-09-02T14:07:47.911517968Z 2022-09-02T14:07:47.911094 E 17 0 10 HeavyDB.cpp:194 Exception: Thrift server exited: Cannot assign requested address: port 6274
2022-09-02T14:07:47.911528142Z 2022-09-02T14:07:47.911118 E 17 0 11 HeavyDB.cpp:194 Exception: Thrift server exited: Cannot assign requested address: port 6276

docker-compose.yml.txt (5.3 KB)
heavy.conf.txt (119 Bytes)
heavyai.conf.txt (119 Bytes)
servers.json.txt (38 Bytes)

I marked the preceding post a Solution, because the original issue (missing Jupyter button) is solved, by setting the new parameter role and parameter value in docker-compose.yml and granting that to the user:

      OMNISCI_JUPYTER_ROLE: omnisci_jupyter

and using the " docker-compose.yml.revised.txt"

I will open a new thread for the different error “[Errno 99] Cannot assign requested address: port xxxx”.

Thanks Neill.

could you just explain better what does it mean " overriding the default start command " , please?

Would I have to change something else respect to what is reported into the documentation, or I missed something already reported here ?

Could you point me to where are documented the details about “overriding the default start command” ?