Skip to content

Django apps NamespacePath support ? #2523

Closed
@douglarek

Description

@douglarek

As Django docs (https://docs.djangoproject.com/en/1.7/ref/applications/#namespace-packages-as-apps-python-3-3) said, Django supports a _NamespacePath app.

But when I used _NamespacePath with django-celery, I got some errors:

/Users/dj/.virtualenvs/3.4.3/lib/python3.4/site-packages/djcelery/loaders.py in find_related_module(app, related_name)
    197 
    198     try:
--> 199         imp.find_module(related_name, app_path)
    200     except ImportError:
    201         return

/Users/dj/.virtualenvs/3.4.3/lib/python3.4/imp.py in find_module(name, path)
    269         # Backwards-compatibility
    270         raise RuntimeError("'list' must be None or a list, "
--> 271                            "not {}".format(type(name)))
    272 
    273     if path is None:

RuntimeError: 'list' must be None or a list, not <class 'str'>

I know celery itself also not support it, will support ?

Thanks

Activity

kmollee

kmollee commented on Mar 13, 2015

@kmollee

This error will occur when use autodiscover_task in celery and one of these app's folder did't have __init__.py file

pretty hard to find out what is going on.. debug information is lack

aztrock

aztrock commented on Jan 9, 2016

@aztrock

Traceback (most recent call last):
File "/opt/kibox/bin/celery", line 11, in
sys.exit(main())
File "/opt/kibox/local/lib/python3.4/site-packages/celery/main.py", line 30, in main
main()
File "/opt/kibox/local/lib/python3.4/site-packages/celery/bin/celery.py", line 81, in main
cmd.execute_from_commandline(argv)
File "/opt/kibox/local/lib/python3.4/site-packages/celery/bin/celery.py", line 770, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/opt/kibox/local/lib/python3.4/site-packages/celery/bin/base.py", line 311, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/opt/kibox/local/lib/python3.4/site-packages/celery/bin/celery.py", line 762, in handle_argv
return self.execute(command, argv)
File "/opt/kibox/local/lib/python3.4/site-packages/celery/bin/celery.py", line 694, in execute
).run_from_argv(self.prog_name, argv[1:], command=argv[0])
File "/opt/kibox/local/lib/python3.4/site-packages/celery/bin/worker.py", line 179, in run_from_argv
return self(_args, *_options)
File "/opt/kibox/local/lib/python3.4/site-packages/celery/bin/base.py", line 274, in call
ret = self.run(_args, *_kwargs)
File "/opt/kibox/local/lib/python3.4/site-packages/celery/bin/worker.py", line 213, in run
).start()
File "/opt/kibox/local/lib/python3.4/site-packages/celery/worker/init.py", line 95, in init
self.app.loader.init_worker()
File "/opt/kibox/local/lib/python3.4/site-packages/celery/loaders/base.py", line 128, in init_worker
self.import_default_modules()
File "/opt/kibox/local/lib/python3.4/site-packages/celery/loaders/base.py", line 116, in import_default_modules
signals.import_modules.send(sender=self.app)
File "/opt/kibox/local/lib/python3.4/site-packages/celery/utils/dispatch/signal.py", line 166, in send
response = receiver(signal=self, sender=sender, *_named)
File "/opt/kibox/local/lib/python3.4/site-packages/amqp/utils.py", line 42, in call
self.set_error_state(exc)
File "/opt/kibox/local/lib/python3.4/site-packages/amqp/utils.py", line 39, in call
*_dict(self.kwargs, **kwargs) if self.kwargs else kwargs
File "/opt/kibox/local/lib/python3.4/site-packages/celery/app/base.py", line 330, in _autodiscover_tasks
self.loader.autodiscover_tasks(packages, related_name)
File "/opt/kibox/local/lib/python3.4/site-packages/celery/loaders/base.py", line 251, in autodiscover_tasks
related_name) if mod)
File "/opt/kibox/local/lib/python3.4/site-packages/celery/loaders/base.py", line 272, in autodiscover_tasks
return [find_related_module(pkg, related_name) for pkg in packages]
File "/opt/kibox/local/lib/python3.4/site-packages/celery/loaders/base.py", line 272, in
return [find_related_module(pkg, related_name) for pkg in packages]
File "/opt/kibox/local/lib/python3.4/site-packages/celery/loaders/base.py", line 294, in find_related_module
_imp.find_module(related_name, pkg_path)
File "/opt/kibox/lib/python3.4/imp.py", line 271, in find_module
"not {}".format(type(name)))
RuntimeError: 'list' must be None or a list, not <class 'str'>

the funny thing is that it was working fine, just kill the process kill -9 celery, as I have cast several times, but this time it stopped working

It is a common mistake, it is almost impossible to find, there is no solution?

aztrock

aztrock commented on Jan 9, 2016

@aztrock

a little help, know where to look.
vim /opt/kibox/lib/python3.4/imp.py
line: 271

Backwards-compatibilit

raise RuntimeError("'list' must be None or a list, "
"not {}".format(type(name)))

to

Backwards-compatibilit

print (path, list)
raise RuntimeError("'list' must be None or a list, "
"not {}".format(type(name)))

you will see something like
_NamespacePath(['/opt/kibox/kibox/apps/apprest']) <class 'list'>

rm /opt/kibox/kibox/apps/apprest/init.py

rm /opt/kibox/kibox/apps/apprest/init.pyc

touch /opt/kibox/kibox/apps/apprest/init.py

ask

ask commented on Jan 12, 2016

@ask
Contributor

The development version (celery master branch) uses app,autodiscover_tasks() with an empty list instead of specifying INSTALLED_APPS, and the Django support populates it with apps.get_apps() instead, which may resolve the issue?

aztrock

aztrock commented on May 24, 2016

@aztrock

app.autodiscover_tasks (), if full app.autodiscover_tasks manually ([ "apps.example", "apps.example2"]), works perfect, app.autodiscover_tasks (settings.INSTALLED_APPS), also works well, app.autodiscover_tasks (lambda : settings.INSTALLED_APPS), works well,

The error detecting achieve, is that if one does not have my files, get the error.

apps / examples / *without py files .py

fails to get more information, lack of time and so far we could migrate to django 1.9

rbreu

rbreu commented on Dec 15, 2016

@rbreu

What's the status of this?

I am having the same problem with celery 4.0.1 and Django 1.10.1.

The issue is that Django allows namespace packages (see link in the OP), i.e. packages without __init__.py file, but celery's autodiscover_tasks fails on those packages with the error above.

I've tracked it down to the call of _imp.find_module here. imp.find_module is deprecated since Python 3.3.

I'm not sure what find_module is supposed to be doing that importlib.import_module can't do? Because when I replace the last bit of that code, starting from line 269, with

try:
    return importlib.import_module('{0}.{1}'.format(package, related_name))
 except ImportError:
    return

the loading seems to work fine for me. But as I said, I don't know what imp.find_module is supposed to be covering here (or why there's an import_module after that without catching ImportError), so I might be overlooking something.

rixx

rixx commented on Aug 18, 2017

@rixx
Contributor

I don't think you are overlooking anything here, @rbreu, so I'm creating a PR to fix this. And if the PR is rejected, at least we will have an explanation.

added a commit that references this issue on Feb 26, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @ask@rbreu@douglarek@rixx@kmollee

        Issue actions

          Django apps NamespacePath support ? · Issue #2523 · celery/celery