Can't find cython even though it's installed

General support questions
Post Reply
martinmin
Posts: 11
Joined: 2021/05/20 06:19:06

Can't find cython even though it's installed

Post by martinmin » 2021/06/09 20:00:06

I am on Centos 7 and created an conda python=3.8 environment:

Code: Select all

conda create -n py385 python=3.8.5

Code: Select all

$ which python
~/miniconda3/envs/py385/bin/python
(py385) [congmin@localhost spbo]$ python
Python 3.8.5 (default, Sep  4 2020, 07:30:14) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
I also installed cython in the same environment and checked:

Code: Select all

python -c "from Cython.Compiler.Version import version; print(version)" 
0.29.21 . 
It's all right with 3.8.5 installed. However, when I compile a cython package it failed and it shows that it is using Python 3.7 to compile, instead of python 3.8. What caused this?

Code: Select all

sudo python setup.py build_ext --inplace
Cythonizing sources
Processing token.pyx
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 840, in setuptools_main
    return main(command_line = 1)
  File "/usr/local/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 858, in main
    result = compile(sources, options)
  File "/usr/local/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 780, in compile
    return compile_multiple(source, options)
  File "/usr/local/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 750, in compile_multiple
    context = options.create_context()
  File "/usr/local/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 597, in create_context
    self.cplus, self.language_level, options=self)
  File "/usr/local/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 80, in __init__
    from . import Builtin, CythonScope
  File "/usr/local/lib/python3.7/site-packages/Cython/Compiler/CythonScope.py", line 5, in <module>
    from .UtilityCode import CythonUtilityCode
  File "/usr/local/lib/python3.7/site-packages/Cython/Compiler/UtilityCode.py", line 3, in <module>
    from .TreeFragment import parse_from_strings, StringParseContext
  File "/usr/local/lib/python3.7/site-packages/Cython/Compiler/TreeFragment.py", line 17, in <module>
    from .Visitor import VisitorTransform
  File "Cython/Compiler/Visitor.py", line 12, in init Cython.Compiler.Visitor
  File "/usr/local/lib/python3.7/inspect.py", line 40, in <module>
    import linecache
  File "/usr/local/lib/python3.7/linecache.py", line 11, in <module>
    import tokenize
  File "/usr/local/lib/python3.7/tokenize.py", line 35, in <module>
    from token import *
  File "token.pyx", line 1, in init spacy.tokens.token
    # cython: infer_types=True
  File "/usr/local/lib/python3.7/site-packages/numpy/__init__.py", line 152, in <module>
    from . import random
  File "/usr/local/lib/python3.7/site-packages/numpy/random/__init__.py", line 181, in <module>
    from . import _pickle
  File "/usr/local/lib/python3.7/site-packages/numpy/random/_pickle.py", line 1, in <module>
    from .mtrand import RandomState
  File "_bit_generator.pxd", line 14, in init numpy.random.mtrand
  File "_bit_generator.pyx", line 255, in init numpy.random._bit_generator
AttributeError: type object 'numpy.random._bit_generator.SeedSequence' has no attribute '__reduce_cython__'
Traceback (most recent call last):
  File "/data0/spbo/bin/cythonize.py", line 60, in process_pyx
    ["cython"] + flags + ["-o", tofile, fromfile], env=os.environ
  File "/usr/local/lib/python3.7/subprocess.py", line 323, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/local/lib/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.7/subprocess.py", line 1522, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'cython': 'cython'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data0/spbo/bin/cythonize.py", line 169, in <module>
    run(args.root)
  File "/data0/spbo/bin/cythonize.py", line 158, in run
    process(base, filename, db)
  File "/data0/spbo/bin/cythonize.py", line 124, in process
    preserve_cwd(base, process_pyx, root + ".pyx", root + ".cpp")
  File "/data0/spbo/bin/cythonize.py", line 87, in preserve_cwd
    func(*args)
  File "/data0/spbo/bin/cythonize.py", line 78, in process_pyx
    raise Exception("Cython failed")
Exception: Cython failed
Traceback (most recent call last):
  File "setup.py", line 199, in <module>
    setup_package()
  File "setup.py", line 187, in setup_package
    generate_cython(root, "spacy")
  File "setup.py", line 114, in generate_cython
    raise RuntimeError("Running cythonize failed")
RuntimeError: Running cythonize failed

martinmin
Posts: 11
Joined: 2021/05/20 06:19:06

Re: Can't find cython even though it's installed

Post by martinmin » 2021/06/09 20:22:23

What's strange is that, it is using python 3.7, not the python 3.8 which my environment has installed.

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Can't find cython even though it's installed

Post by TrevorH » 2021/06/09 21:02:35

sudo strips lots of environment variables and substitutes others.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

martinmin
Posts: 11
Joined: 2021/05/20 06:19:06

Re: Can't find cython even though it's installed

Post by martinmin » 2021/06/09 22:43:00

I didn't get. What do you mean? Please be a bit more specific and thank you.

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Can't find cython even though it's installed

Post by TrevorH » 2021/06/09 22:49:10

Lots of things that you had set as the user you used will not have been passed through via sudo to run `sudo python setup.py build_ext --inplace` - specifically $PATH for a start. So when it went to run 'python' as root (via sudo) it wil lhave searched a different PATH and found a different python executable...
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

martinmin
Posts: 11
Joined: 2021/05/20 06:19:06

Re: Can't find cython even though it's installed

Post by martinmin » 2021/06/10 00:40:54

So how to solve this issue when using Anaconda? Our typical environment is to run using sudo to install.

martinmin
Posts: 11
Joined: 2021/05/20 06:19:06

Re: Can't find cython even though it's installed

Post by martinmin » 2021/06/10 21:03:22

Even just using Python 3.7, it still complains the absence of cython, as below. There is no Python 3.8 involved. Also, cython is already installed for python 3.7.

Code: Select all

sudo python setup.py build_ext --inplace
Cythonizing sources
Processing token.pyx
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 840, in setuptools_main
    return main(command_line = 1)
  File "/usr/local/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 858, in main
    result = compile(sources, options)
  File "/usr/local/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 780, in compile
    return compile_multiple(source, options)
  File "/usr/local/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 750, in compile_multiple
    context = options.create_context()
  File "/usr/local/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 597, in create_context
    self.cplus, self.language_level, options=self)
  File "/usr/local/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 80, in __init__
    from . import Builtin, CythonScope
  File "/usr/local/lib/python3.7/site-packages/Cython/Compiler/CythonScope.py", line 5, in <module>
    from .UtilityCode import CythonUtilityCode
  File "/usr/local/lib/python3.7/site-packages/Cython/Compiler/UtilityCode.py", line 3, in <module>
    from .TreeFragment import parse_from_strings, StringParseContext
  File "/usr/local/lib/python3.7/site-packages/Cython/Compiler/TreeFragment.py", line 17, in <module>
    from .Visitor import VisitorTransform
  File "Cython/Compiler/Visitor.py", line 12, in init Cython.Compiler.Visitor
  File "/usr/local/lib/python3.7/inspect.py", line 40, in <module>
    import linecache
  File "/usr/local/lib/python3.7/linecache.py", line 11, in <module>
    import tokenize
  File "/usr/local/lib/python3.7/tokenize.py", line 35, in <module>
    from token import *
  File "token.pyx", line 1, in init spacy.tokens.token
    # cython: infer_types=True
  File "/usr/local/lib/python3.7/site-packages/numpy/__init__.py", line 152, in <module>
    from . import random
  File "/usr/local/lib/python3.7/site-packages/numpy/random/__init__.py", line 181, in <module>
    from . import _pickle
  File "/usr/local/lib/python3.7/site-packages/numpy/random/_pickle.py", line 1, in <module>
    from .mtrand import RandomState
  File "_bit_generator.pxd", line 14, in init numpy.random.mtrand
  File "_bit_generator.pyx", line 255, in init numpy.random._bit_generator
AttributeError: type object 'numpy.random._bit_generator.SeedSequence' has no attribute '__reduce_cython__'
Traceback (most recent call last):
  File "/data0/congmin/wb-nlp-tool/spaCy-weibo/bin/cythonize.py", line 60, in process_pyx
    ["cython"] + flags + ["-o", tofile, fromfile], env=os.environ
  File "/usr/local/lib/python3.7/subprocess.py", line 323, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/local/lib/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.7/subprocess.py", line 1522, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'cython': 'cython'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data0/congmin/wb-nlp-tool/spaCy-weibo/bin/cythonize.py", line 169, in <module>
    run(args.root)
  File "/data0/congmin/wb-nlp-tool/spaCy-weibo/bin/cythonize.py", line 158, in run
    process(base, filename, db)
  File "/data0/congmin/wb-nlp-tool/spaCy-weibo/bin/cythonize.py", line 124, in process
    preserve_cwd(base, process_pyx, root + ".pyx", root + ".cpp")
  File "/data0/congmin/wb-nlp-tool/spaCy-weibo/bin/cythonize.py", line 87, in preserve_cwd
    func(*args)
  File "/data0/congmin/wb-nlp-tool/spaCy-weibo/bin/cythonize.py", line 78, in process_pyx
    raise Exception("Cython failed")
Exception: Cython failed
Traceback (most recent call last):
  File "setup.py", line 199, in <module>
    setup_package()
  File "setup.py", line 187, in setup_package
    generate_cython(root, "spacy")
  File "setup.py", line 114, in generate_cython
    raise RuntimeError("Running cythonize failed")
RuntimeError: Running cythonize failed

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Can't find cython even though it's installed

Post by jlehtone » 2021/06/10 21:40:00

martinmin wrote:
2021/06/10 00:40:54
So how to solve this issue when using Anaconda? Our typical environment is to run using sudo to install.
Our typical environment is to install without sudo. Well, properly RPM-packaged packets are installed with elevated rights, but all "source", "binary blob", "pip/setup", "R packages", etc are installed as regular user to directories, where regular user can write to. Definitely not to system directories.

I have not yet decided whether I loathe or just strongly dislike 'conda'. Doesn't conda effectively create python virtual env? Why would you need sudo with it?

martinmin
Posts: 11
Joined: 2021/05/20 06:19:06

Re: Can't find cython even though it's installed

Post by martinmin » 2021/06/11 01:57:31

I need 'sudo' because our machines are configured by op people this way. If you need write permission, you need to use sudo.

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Can't find cython even though it's installed

Post by jlehtone » 2021/06/12 17:38:56

martinmin wrote:
2021/06/11 01:57:31
I need 'sudo' because our machines are configured by op people this way. If you need write permission, you need to use sudo.
Technically, couldn't you merely use sudo to create directory that has permissions for your accout and then run install without sudo?

Post Reply