weixin_39620943
2020-12-30 22:44 阅读 2

(sql-load) not working using python scheme wrapper scheme_eval_h

I am trying to load atoms into an atomspace from python using scheme_eval_h. The problem I am facing is that I am not getting any response from sql-load.

The project is running inside Docker, where the python app and Postgres are running as services. When trying to load the atoms from a guile shell inside the docker container that has the python app, It was successful.

bash
enku-1604-xenial-64-minimal$ docker exec -it python_app_docker /bin/bash
root:~/mozi# guile
;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
;;;       or pass the --no-auto-compile argument to disable.
;;; compiling /usr/local/share/opencog/scm/opencog.scm
;;; compiled /root/.cache/guile/ccache/2.2-LE-8-3.A/usr/local/share/opencog/scm/opencog.scm.go
GNU Guile 2.2.3
Copyright (C) 1995-2017 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
sheme@(guile-user)> (sql-open "postgres://username:password/database")
NOTICE:  relation "uuid_pool" already exists, skipping
NOTICE:  relation "vuid_pool" already exists, skipping
$1 = ()
scheme@(guile-user)> (sql-load)
Loading all atoms; maxuuid=704384 max height=10
Loading all atoms: Max Height is 10 stepsize=12470 chunks=57
        Loaded 100K atoms in 7 seconds (14285 per second).
Loaded 156515 atoms at height 0
        Loaded 200K atoms in 13 seconds (15384 per second).
        Loaded 300K atoms in 15 seconds (20000 per second).
        Loaded 400K atoms in 18 seconds (22222 per second).
        Loaded 500K atoms in 20 seconds (25000 per second).
        Loaded 600K atoms in 23 seconds (26086 per second).
        Loaded 700K atoms in 26 seconds (26923 per second).
Loaded 545154 atoms at height 1
Loaded 823 atoms at height 2
Loaded 782 atoms at height 3
Loaded 228 atoms at height 4
Loaded 161 atoms at height 5
Loaded 79 atoms at height 6
Loaded 62 atoms at height 7
Loaded 76 atoms at height 8
Loaded 150 atoms at height 9
Loaded 11 atoms at height 10
Finished loading 704041 atoms in total in 28 seconds (25144 per second)
$2 = ()
scheme@(guile-user)> 

However, when running the python app, there is no response. This is the python file containing load_atomspace() function which is called when starting the flask application.

python
import os
from opencog.atomspace import AtomSpace
from opencog.scheme_wrapper import scheme_eval_h

from config import PROJECT_ROOT
guile_path = os.path.join(PROJECT_ROOT, ".guile")

def load_atomspace():
    atomspace = AtomSpace()
    scheme_eval_h(atomspace, '(load "{}")'.format(guile_path))
    scheme_eval_h(atomspace, '(sql-open "postgres://username:password/database")')
    scheme_eval_h(atomspace, '(sql-load)')

This is the log from the python app. It logs relation "uuid_pool" already exists, skipping which shows it has connected to the Postgres database. However after that there is no response.

bash
[2018-08-17 09:38:32 +0000] [23] [INFO] Starting gunicorn 19.9.0
[2018-08-17 09:38:32 +0000] [23] [INFO] Listening at: http://0.0.0.0:5000 (23)
[2018-08-17 09:38:32 +0000] [23] [INFO] Using worker: eventlet
[2018-08-17 09:38:32 +0000] [26] [INFO] Booting worker with pid: 26
NOTICE:  relation "uuid_pool" already exists, skipping
NOTICE:  Set uuid_pool sequence to 703985
NOTICE:  relation "vuid_pool" already exists, skipping

该提问来源于开源项目:opencog/atomspace

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享

14条回答 默认 最新

  • weixin_39624700 weixin_39624700 2020-12-30 22:44

    No clue. ??

    点赞 评论 复制链接分享
  • weixin_39624700 weixin_39624700 2020-12-30 22:44

    The only way I know how to debug this is to insert print statements into strategic locations, and try to narrow down the first one that doesn't print. The vuid_pool message shows that, indeed, the python code is able to get through guile, and into the database code, and run.

    • Perhaps (sql-open) never returned?
    • Perhaps there are multiple versions of the atomspace shared libraries installed, and python is using the wrong ones?
    • Perhaps it worked, but the prints statements are just not becoming visible?

    Those are things I can guess at...

    点赞 评论 复制链接分享
  • weixin_39878716 weixin_39878716 2020-12-30 22:44

    I have no fast answer, the only way I see is reproducing and debugging it.

    点赞 评论 复制链接分享
  • weixin_39620943 weixin_39620943 2020-12-30 22:44

    Upon further investigation, I have found the following.

    • Perhaps (sql-open) never returned?

    (sql-open) has returned and the following output is the proof as (sql-open) prints this statement from the guile shell as well.

    bash
    NOTICE:  relation "uuid_pool" already exists, skipping
     NOTICE:  Set uuid_pool sequence to 703985
     NOTICE:  relation "vuid_pool" already exists, skipping
    
    • Perhaps there are multiple versions of the atomspace shared libraries installed, and python is using the wrong ones?

    Our Docker is based off the opencog/opencog-dev docker, which has the latest builds of opencog, atomspace and cogutils.

    • Perhaps it worked, but the prints statements are just not becoming visible?

    It hasn't worked as I'll explain below.

    When loading dependencies from a guile file there is no output, meaning (sql-load) is not working. However when loading only opencog persistence dependencies using scheme_eval_h like as follows

    python
    def load_atomspace():
        atomspace = AtomSpace()
        scheme_eval_h(atomspace,'''
             (use-modules (opencog)
                          (opencog persist) (opencog persist-sql))
             ''')
        scheme_eval_h(atomspace, '(sql-open "postgres://username:password/database")')
        scheme_eval_h(atomspace, '(sql-load)')
    

    it tries loading the atoms by outputing the following,

    bash
    mozi_1      | Loading all atoms; maxuuid=190156 max height=0
    mozi_1      | Loading all atoms: Max Height is 0 stepsize=10756 chunks=18
    

    However throws the following error when trying to load the atoms. The error is explainable as we haven't loaded the NLP modules.

    bash
    mozi_1      | terminate called after throwing an instance of 'opencog::InvalidParamException'
    mozi_1      |   what():  Not a valid typename: 'SentenceNode' (/tmp/atomspace-master/opencog/atoms/core/TypeNode.h:56)
    

    My suspiscion is that there is a module being loaded in the guile file that might be interfering with the workings of (sql-load). The guile file we are loading is as follows.

    scheme
    (use-modules (ice-9 readline))
    (activate-readline)
    (add-to-load-path "/usr/local/share/opencog/scm")
    (add-to-load-path "/usr/local/share/opencog/scm/opencog")
    (add-to-load-path ".")
    (use-modules (opencog query))
    (use-modules (opencog exec) (opencog bioscience))
    (use-modules (opencog)
                 (opencog nlp)
                 (opencog nlp relex2logic)
                 (opencog openpsi)
                 (opencog attention)
                 (opencog ghost)
                 (opencog ghost procedures))
    
    (use-modules (opencog)
             (opencog persist) (opencog persist-sql))
    
    (use-modules (opencog python))
    (use-modules (srfi srfi-1))
    (use-modules (ice-9 regex))
    (use-modules (rnrs base) (rnrs exceptions))
    (set-relex-server-host)
    (use-modules (json))
    

    We are loading the guile file as follows from python.

    python
    from opencog.atomspace import AtomSpace
    from opencog.scheme_wrapper import scheme_eval_h
    
    from config import PROJECT_ROOT
    guile_path = os.path.join(PROJECT_ROOT, ".guile")
    def load_atomspace():
        atomspace = AtomSpace()
        scheme_eval_h(atomspace, '(load "{}")'.format(guile_path))
     
    点赞 评论 复制链接分享
  • weixin_39624700 weixin_39624700 2020-12-30 22:44

    First some preliminary comments:

    
    (use-modules (ice-9 readline))
    (activate-readline)
    

    You don't need readline, if you are not using the command-line. Readline just makes the up and down arrows work, and nothing else.

    
    (add-to-load-path "/usr/local/share/opencog/scm")
    (add-to-load-path "/usr/local/share/opencog/scm/opencog")
    (add-to-load-path ".")
    

    The last two are not needed. Adding "." is dangerous.

    
    (use-modules (opencog query))
    

    You should (use-modules (opencog)) before any of the other opencog modules.

    
    (use-modules (opencog exec) (opencog bioscience))
    (use-modules (opencog)
                 (opencog nlp)
                 (opencog nlp relex2logic)
                 (opencog openpsi)
                 (opencog attention)
                 (opencog ghost)
                 (opencog ghost procedures))
    (use-modules (opencog)
    

    You do not need to say (use-modules (opencog)) a second time....

    
             (opencog persist) (opencog persist-sql))
    
    (use-modules (opencog python))
    (use-modules (srfi srfi-1))
    (use-modules (ice-9 regex))
    (use-modules (rnrs base) (rnrs exceptions))
    

    You probably do not need srfi-i, ice-9 or rnrs ....!!??

    
    (set-relex-server-host)
    (use-modules (json))
    

    Do you need json ?? For just hacking things at the command line, its harmless to load all this stuff. But for an automated system, this is all probably too much.

    点赞 评论 复制链接分享
  • weixin_39624700 weixin_39624700 2020-12-30 22:44

    For this one:

    
    guile_path = os.path.join(PROJECT_ROOT, ".guile")
    

    It would be much better to rename .guile to mozi-setup.scm and load that. The point is that .guile is a hidden user file, that users hack on to add custom commands; you should not be overloading or hacking on user files to store configs.

    点赞 评论 复制链接分享
  • weixin_39624700 weixin_39624700 2020-12-30 22:44
    
     scheme_eval_h(atomspace, '(load "{}")'.format(guile_path))
    

    Use scheme_eval instead of scheme_eval_h. The former just runs some scheme; the later returns an atom handle. Since (load) does not return an atom handle, using scheme_eval_h is pointless (well, and harmless -- it should not hurt, not really)

    None of the suggestions above are likely to fix the bug; they are simply "good practice" items.

    点赞 评论 复制链接分享
  • weixin_39624700 weixin_39624700 2020-12-30 22:44

    I cannot reproduce this bug on my setup.

    When you say:

    However throws the following error when trying to load the atoms. The error is explainable as we haven't loaded the NLP modules.

    I don't understand ... in your .guile, you do load the NLP modules, so that should be OK. If you don't, then yes, I suppose some error is thrown. Maybe the error path could be cleaned up? The system should not hang or crash; it should report an error to the user.

    C++ exceptions are supposed to be turned into python exceptions, so that you can catch them. Maybe the C++-exception to python-exception converter code is broken ...

    点赞 评论 复制链接分享
  • weixin_39620943 weixin_39620943 2020-12-30 22:44

    Your "good practice" items have worked and I can see logs now. The guile file as per your comments is as follows.

    scheme
    (add-to-load-path "/usr/local/share/opencog/scm")
    (add-to-load-path "/usr/local/share/opencog/scm/opencog")
    (add-to-load-path ".")
    (use-modules (opencog) 
                 (opencog query)
                 (opencog exec) 
                 (opencog bioscience)
                 (opencog nlp)
                 (opencog nlp relex2logic)
                 (opencog openpsi)
                 (opencog attention)
                 (opencog ghost)
                 (opencog ghost procedures)
                 (opencog persist) 
                 (opencog persist-sql)
                 (opencog python)
                 ;;(srfi srfi-1)
                 (ice-9 regex)
                 (rnrs base) 
                 (rnrs exceptions))
    (set-relex-server-host)
    (use-modules (json))
    

    And when I load this guile file this is the error I get. But it has started loading the atoms.

    bash
    NOTICE:  relation "uuid_pool" already exists, skipping
    mozi_1      | NOTICE:  Set uuid_pool sequence to 189757
    mozi_1      | NOTICE:  relation "vuid_pool" already exists, skipping
    mozi_1      | BFD: /lib/x86_64-linux-gnu/libpthread.so.0: warning: loop in section dependencies detected
    mozi_1      | Loading all atoms; maxuuid=190156 max height=0
    mozi_1      | Loading all atoms: Max Height is 0 stepsize=10756 chunks=18
    mozi_1      | BFD: /usr/lib/x86_64-linux-gnu/libgomp.so.1: warning: loop in section dependencies detected
    mozi_1      | BFD: /usr/lib/x86_64-linux-gnu/libgomp.so.1: warning: loop in section dependencies detected
    

    The (json) module is required for our specific "ghost" needs. You can check the repo here. https://github.com/Habush/guile-json

    点赞 评论 复制链接分享
  • weixin_39624700 weixin_39624700 2020-12-30 22:44

    The uuid messages are annoying but harmless. It would take some work to make them go away.

    The BFD warnings are annoying and harmless. Its really really hard to make them go away.

    ... so, it has started loading the atoms ... has it finished? Based on your earlier error report, I'm pretty sure that the c++exception to python-exception converter is broken.

    点赞 评论 复制链接分享
  • weixin_39620943 weixin_39620943 2020-12-30 22:44

    It has started loading the atoms but it crashes after reporting loop in section dependencies. We are using gunicorn as our server and it reboots the worker thread which starts the server after the warning then tries to load the atoms again and the same warning comes up. Any ideas?

    I believe the warning is actually harmful and causing the problem. What do you think?

    点赞 评论 复制链接分享
  • weixin_39624700 weixin_39624700 2020-12-30 22:44

    Yes, I'm starting to think you might be right. If I google libgomp to see what it is, I see that it is a low-level library involved in multi-processing and runtime optimizations. Its very different than libbfd ... or more like the starship version of libbfd. So the BFD error message had me confused, but I'm thinking you might be right.

    This presents a new problem: libgomp is rather low-level, and tricking it into not crashing/exiting-on-warning might be hard. Two suggestions:

    1) read more about libgomp. Maybe "exit-on-warning" is a default, and there is some flag that over-rides this? A linker flag, r.g. -Wl,dont-crash-on-warning or something like that. Or maybe -Wl,disable-libgomp-optimization ... there are hundreds of these flags.

    2) Maybe compiling source with -O2 optimization, instead of the current -O3 will make this go away? Some fine-grains -Wl flag would be better.

    点赞 评论 复制链接分享
  • weixin_39624700 weixin_39624700 2020-12-30 22:44

    Have you looked at the stack traces? Run it under GDB, see what it prints.

    Stack traces can sometimes be very confusing, and sometimes very useful. I'm guessing that, in this case, it will be very confusing. But still, take a look.

    点赞 评论 复制链接分享
  • weixin_39624700 weixin_39624700 2020-12-30 22:44

    Is this still an issue? The accumulated comments and history make this hard to read and understand... I'm closing this due to lack of progress, if it happens again, please open a new bug.

    点赞 评论 复制链接分享

相关推荐