Skip to content

Numba warnings #252

Closed
Closed
@mbr085

Description

@mbr085

First I want to thank you for sharing this great work. It is very useful for me.
The first time I execute UMAP fit and transform I get some numba warnings. They do not seem to be critical, but they are somewhat disturbing.

numba.__version__
'0.44.0'

sys.version
'3.7.3 (default, Mar 27 2019, 22:11:17) \n[GCC 7.3.0]'

reducer = umap.UMAP(random_state=42)
reducer.fit(X)
Y = reducer.transform(X)
/home/mbr085/anaconda3/envs/divisivegater/lib/python3.7/site-packages/umap/umap_.py:349: NumbaWarning: 
Compilation is falling back to object mode WITH looplifting enabled because Function "fuzzy_simplicial_set" failed type inference due to: Untyped global name 'nearest_neighbors': cannot determine Numba type of <class 'function'>

File "../../../../../anaconda3/envs/divisivegater/lib/python3.7/site-packages/umap/umap_.py", line 467:
def fuzzy_simplicial_set(
    <source elided>
    if knn_indices is None or knn_dists is None:
        knn_indices, knn_dists, _ = nearest_neighbors(
        ^

  @numba.jit()
/home/mbr085/anaconda3/envs/divisivegater/lib/python3.7/site-packages/numba/compiler.py:725: NumbaWarning: Function "fuzzy_simplicial_set" was compiled in object mode without forceobj=True.

File "../../../../../anaconda3/envs/divisivegater/lib/python3.7/site-packages/umap/umap_.py", line 350:
@numba.jit()
def fuzzy_simplicial_set(
^

  self.func_ir.loc))
/home/mbr085/anaconda3/envs/divisivegater/lib/python3.7/site-packages/numba/compiler.py:734: NumbaDeprecationWarning: 
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

File "../../../../../anaconda3/envs/divisivegater/lib/python3.7/site-packages/umap/umap_.py", line 350:
@numba.jit()
def fuzzy_simplicial_set(
^

  warnings.warn(errors.NumbaDeprecationWarning(msg, self.func_ir.loc))

Activity

lmcinnes

lmcinnes commented on Jun 13, 2019

@lmcinnes
Owner

These are, indeed, non-critical -- numba has just gotten a little more verbose about it's warnings, and umap continues to be somewhat optimistic about how well numba can compile things. In this case numba is just letting you know that umap was a little over-optimistic and not as much could be as completely compiled as one might have hoped. I'll try to fix this up in an upcoming release. Thanks for the heads up, it is appreciated.

00krishna

00krishna commented on Jun 26, 2019

@00krishna

I was getting the same warnings. I added the following code to my notebooks and it solved it.

import warnings
warnings.filterwarnings('ignore')

This was based upon the StackExchange Post
https://stackoverflow.com/questions/9031783/hide-all-warnings-in-ipython

Kimi71Jedidiah

Kimi71Jedidiah commented on Jul 13, 2019

@Kimi71Jedidiah

I came across the same problem!
S.umap(max_pc=max_pc,n_neighbors=k,min_dist=0.05)
微信图片_20190713072357

lmcinnes

lmcinnes commented on Jul 14, 2019

@lmcinnes
Owner

This is actually fine -- the latest version of numba now issues warning for routines it couldn't completely compile. There will be a release of umap-learn soon that will remove the ineffective jit instructions and thus remove the warnings.

sjfleming

sjfleming commented on Aug 2, 2019

@sjfleming

Could you not do
@numba.jit(forceobj=True) ?

lmcinnes

lmcinnes commented on Aug 5, 2019

@lmcinnes
Owner

That might we work yes. Right now it is mostly just a matter of having the time to sit down and go through all the options and the code and make changes. I was very busy last month, and am on vacation right now and trying not to look at code. Hopefully I can make some time in a couple of weeks.

sleighsoft

sleighsoft commented on Aug 8, 2019

@sleighsoft
Collaborator

Enjoy your vacation :)

To help you resolve the warnings quickly I compiled a list of all numba errors.

A list of warnings I currently see:

The keyword argument 'parallel=True' was specified but no transformation for 
parallel execution was possible.

To find out why, try turning on parallel diagnostics, see
http://numba.pydata.org/numba-doc/latest/user/parallel.html#diagnostics for
help.File "../umap/nndescent.py", line 47:   
    @numba.njit(parallel=True)
    def nn_descent(
Compilation is falling back to object mode WITH looplifting enabled because Function 
"fuzzy_simplicial_set" failed type inference due to: Untyped global name 
'nearest_neighbors': cannot determine Numba type o$ <class 'function'>
File "../umap/umap_.py", line 467:
    def fuzzy_simplicial_set(
    <source elided>
    if knn_indices is None or knn_dists is None:
        knn_indices, knn_dists, _ = nearest_neighbors(
        ^
NumbaWarning: Function "fuzzy_simplicial_set" was compiled in object mode without 
forceobj=True. 
File "../umap/umap_.py", line 350:
@numba.jit()
def fuzzy_simplicial_set(
^
The keyword argument 'parallel=True' was specified but no transformation for
parallel execution was possible. To find out why, try turning on parallel diagnostics,
see http://numba.pydata.org/numba-doc/latest/user/parallel.html#diagnostics for help.
File "../umap/utils.py", line 409:
@numba.njit(parallel=True)
def build_candidates(current_graph, n_vertices, n_neighbors, max_candidates, rng_state):
^
umap/rp_tree.py:450: NumbaWarning:  Compilation is falling back to object mode 
WITH looplifting enabled because Function "make_euclidean_tree" failed type 
inference due to: Cannot unify RandomProjectionTreeNode(array(int64, 1d, C), 
bool, none, none, none, none) and RandomProjectionTreeNode(none, bool, 
array(float32, 1d, C), float64, RandomProjectionTreeNode(array(int64, 1d, C), bool, 
none, none, none, none), RandomProjectionTreeNode(array(int64, 1d, 
C), bool, none, none, none, none)) for '$14.16', defined at ../umap/rp_tree.py (457)  
                                                                                               
File "../umap/rp_tree.py", line 457:            
def make_euclidean_tree(data, indices, rng_state, leaf_size=30):  
<source elided>                              
        left_node = make_euclidean_tree(data, left_indices, rng_state, leaf_size)
        ^
                                                                                              
[1] During: resolving callee type: recursive(type(CPUDispatcher(
<function make_euclidean_tree at 0x7fafae56a378>)))
[2] During: typing of call at ../umap/rp_tree.py (457)   
                                                                                                                
File "../umap/rp_tree.py", line 457:       
def make_euclidean_tree(data, indices, rng_state, leaf_size=30):
    <source elided>
    left_node = make_euclidean_tree(data, left_indices, rng_state, leaf_size)
    ^ 
@numba.jit()
NumbaWarning: Function "make_euclidean_tree" was compiled in object mode without 
forceobj=True.       
sleighsoft

sleighsoft commented on Aug 14, 2019

@sleighsoft
Collaborator

Changed in: afb819e

sleighsoft

sleighsoft commented on Sep 5, 2019

@sleighsoft
Collaborator

Should be resolved in 0.3.10 release 0.4

EDIT:
This warning is caused by umap-learn package and is just that. A warning that numba, a package used to speed up code execution by JIT compiling code, could not optimize the code that was annotated to be JIT compiled.

This is 100% my bad. I thought @lmcinnes added this to 0.3.10 but I think it only made it into the upcoming release of 0.4.

The PR for the changes is here #287

asfix

asfix commented on Oct 16, 2019

@asfix

The keyword argument 'parallel=True' was specified but no transformation for
parallel execution was possible.

To find out why, try turning on parallel diagnostics, see
http://numba.pydata.org/numba-doc/latest/user/parallel.html#diagnostics for
help.File "../umap/nndescent.py", line 47:
@numba.njit(parallel=True)
def nn_descent(

I am having this problem too..

/usr/local/lib/python3.6/dist-packages/umap/spectral.py:229: UserWarning: Embedding a total of 178 separate connected components using meta-embedding (experimental)
n_components
/usr/local/lib/python3.6/dist-packages/sklearn/metrics/pairwise.py:1575: DataConversionWarning: Data was converted to boolean for metric jaccard
warnings.warn(msg, DataConversionWarning)

When I press a key, I started to get another message. Would not I run my script and obtain reduced dimensions?

lmcinnes

lmcinnes commented on Oct 16, 2019

@lmcinnes
Owner

Sorry @asfix ; these are just warnings and don't represent a problem beyond the fact that various things have been done in the processing that a user may wish to be aware of. It should all run without problem, but there may be some warning messages flashing by.

suluxan

suluxan commented on Oct 31, 2019

@suluxan

Hey,
We are getting this same error but the warnings are causing our pipeline to fail rather than just skip over them. Is there any way to ignore warnings? Or has this been addressed?
i.e.

server_1    | [0] stderr: /usr/local/lib/python2.7/dist-packages/numba/compiler.py:602: NumbaPerformanceWarning:
server_1    | [0] The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.
server_1    | [0]
server_1    | [0] To find out why, try turning on parallel diagnostics, see http://numba.pydata.org/numba-doc/latest/user/parallel.html#diagnostics for help.
server_1    | [0]
server_1    | [0] File "../usr/local/lib/python2.7/dist-packages/umap/nndescent.py", line 47:
server_1    | [0]     @numba.njit(parallel=True)
server_1    | [0]     def nn_descent(
server_1    | [0]     ^
server_1    | [0]
server_1    | [0]   self.func_ir.loc))
server_1    | [0]
server_1    | [0] stderr: UMAP(a=None, angular_rp_forest=False, b=None, init='spectral',
server_1    | [0]    learning_rate=1.0, local_connectivity=1, metric='correlation',
server_1    | [0]    metric_kwds=None, min_dist=0.3, n_components=2, n_epochs=None,
server_1    | [0]    n_neighbors=30, negative_sample_rate=5, random_state=None,
server_1    | [0]    repulsion_strength=1.0, set_op_mix_ratio=1.0, spread=1.0,
server_1    | [0]    target_metric='categorical', target_metric_kwds=None,
server_1    | [0]    target_n_neighbors=-1, target_weight=0.5, transform_queue_size=4.0,
server_1    | [0]    transform_seed=42, verbose=True)
sleighsoft

sleighsoft commented on Nov 1, 2019

@sleighsoft
Collaborator

It has been addressed in 0.4dev. Could you try with that branch, please. And report back any issues.

suluxan

suluxan commented on Nov 14, 2019

@suluxan

Is the 0.4dev only available for python 3.5?

sleighsoft

sleighsoft commented on Nov 19, 2019

@sleighsoft
Collaborator

If I remember correctly there was a PR that included the @ operator which is only available in Python 3.5.
I cannot find the PR but I assume that when you don't use that code path you are fine to use a version lower than 3.5

lmcinnes

lmcinnes commented on Nov 20, 2019

@lmcinnes
Owner

I believe future releases will be targeting 3.5 and upwards. So things may work for now, but i a version or so you'll likely need a newer python if you haven't gotten onto 3.5 or newer.

suluxan

suluxan commented on Nov 20, 2019

@suluxan

Yeah there was build problems on python 2.7 due to the @ but we've updated to 3.7.3 and everything is smooth now! Thank you very much

moldach

moldach commented on Jan 28, 2020

@moldach

This warning is caused by umap-learn package and is just that. A warning that numba, a package used to speed up code execution by JIT compiling code, could not optimize the code that was annotated to be JIT compiled.

I'm seeking clarity on: The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.

I'm trying to do some benchmarking and would like to know if this means that UMAP only ran on one core?

Just found issue #317 from November 2019 so I'm guessing parallelization is still not available out-of-the-box?

sleighsoft

sleighsoft commented on Feb 2, 2020

@sleighsoft
Collaborator

This can just mean that a single function within UMAP could not be optimized to be run in parallel by Numba. It does not mean that everything else, that can be, wasn't executed in parallel. This should not happen with the current dev branch. If so, please report.

naity

naity commented on Feb 4, 2020

@naity

I use the following to ignore the numba warning without affecting other warnings for now.

# silence NumbaPerformanceWarning
import warnings
from numba.errors import NumbaPerformanceWarning

warnings.filterwarnings("ignore", category=NumbaPerformanceWarning)
scvannost

scvannost commented on Feb 8, 2020

@scvannost

This solution works for me!

added
Good ReadsIssues that discuss important topics regarding UMAP, that provide useful code or nice visualizations
on Feb 9, 2020
vasiliskatr

vasiliskatr commented on Mar 23, 2021

@vasiliskatr

Thank you for numba!

This warning: "The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible."

is always occurring only on the 1st execution of the jited function. Every subsequent execution runs smoothly and and several times faster. Is there a way to avoid this warning entirely?

Many computationally-heavy functions are called only once and thus it would be a great improvement if it could be avoided.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Good ReadsIssues that discuss important topics regarding UMAP, that provide useful code or nice visualizations

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @asfix@00krishna@moldach@mbr085@scvannost

        Issue actions

          Numba warnings · Issue #252 · lmcinnes/umap