Celery CRITICAL/MainProcess] Unrecoverable error: AttributeError(“'float' object has no attribute 'items'”,)

  • A+
Category:Languages

I've been running a flask application with a celery worker and redis in three separated docker containers without any issue.

This is how I start it:

celery worker -A app.controller.engine.celery -l info --concurrency=2 --pool eventlet

Celery starts fine:

 -------------- celery@a828bd5b0089 v4.2.1 (windowlicker) ---- **** ----- --- * ***  * -- Linux-4.9.93-linuxkit-aufs-x86_64-with 2018-11-15 16:06:59 -- * - **** --- - ** ---------- [config] - ** ---------- .> app:         app.controller.engine:0x7f8ba4eb70b8 - ** ---------- .> transport:   redis://redis:6379/0 - ** ---------- .> results:     redis://redis:6379/1 - *** --- * --- .> concurrency: 2 (eventlet) -- ******* ---- .> task events: ON --- ***** -----  -------------- [queues]                 .> celery           exchange=celery(direct) key=celery   [tasks]   . app.controller.engine.do  INFO:engineio:Server initialized for eventlet. INFO:engineio:Server initialized for threading. [2018-11-15 15:44:34,301: INFO/MainProcess] Connected to redis://redis:6379/0 [2018-11-15 15:44:34,321: INFO/MainProcess] mingle: searching for neighbors [2018-11-15 15:44:35,358: INFO/MainProcess] mingle: all alone [2018-11-15 15:44:35,396: INFO/MainProcess] pidbox: Connected to redis://redis:6379/0. [2018-11-15 15:44:35,415: INFO/MainProcess] celery@12af03844cd0 ready. 

But when starting a task , it suddenly crashes:

[2018-11-15 15:02:03,224: CRITICAL/MainProcess] Unrecoverable error: AttributeError("'float' object has no attribute 'items'",) Traceback (most recent call last):   File "/app/env/lib/python3.6/site-packages/celery/worker/worker.py", line 205, in start     self.blueprint.start(self)   File "/app/env/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start     step.start(parent)   File "/app/env/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start     return self.obj.start()   File "/app/env/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 317, in start     blueprint.start(self)   File "/app/env/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start     step.start(parent)   File "/app/env/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 593, in start     c.loop(*c.loop_args())   File "/app/env/lib/python3.6/site-packages/celery/worker/loops.py", line 121, in synloop     connection.drain_events(timeout=2.0)   File "/app/env/lib/python3.6/site-packages/kombu/connection.py", line 301, in drain_events     return self.transport.drain_events(self.connection, **kwargs)   File "/app/env/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 963, in drain_events     get(self._deliver, timeout=timeout)   File "/app/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 359, in get     ret = self.handle_event(fileno, event)   File "/app/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 341, in handle_event     return self.on_readable(fileno), self   File "/app/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 337, in on_readable     chan.handlers[type]()   File "/app/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 724, in _brpop_read     self.connection._deliver(loads(bytes_to_str(item)), dest)   File "/app/env/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 983, in _deliver     callback(message)   File "/app/env/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 632, in _callback     self.qos.append(message, message.delivery_tag)   File "/app/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 149, in append     pipe.zadd(self.unacked_index_key, time(), delivery_tag) /   File "/app/env/lib/python3.6/site-packages/redis/client.py", line 2263, in zadd     for pair in iteritems(mapping):   File "/app/env/lib/python3.6/site-packages/redis/_compat.py", line 123, in iteritems     return iter(x.items()) AttributeError: 'float' object has no attribute 'items' 

I don't see any reference to my code in the stacktrace.

This is the task method:

@celery.task() def do(module_name, json_input):     logger.info('____ Running _________________________')     logger.info('testing ***********************')     #modules.run(module_name, json_input) 

I commented the call to my actual code just to verify I didn't mess it up with my modules code.. but it's crashing anyways.

Anybody has an idea what could be wrong or how to debug it properly?

Thanks a lot

 


The problem was that the my celery container downloaded a new release of the redis package with pip at build time, eventually with a bug or a serious change that crashes when celery tries to use it when connecting to redis. This new redis library is the 3.0.0 and it was released today.

To fix the issue you have to specify the previous version in the requirements.txt (which is 2.10.6).

And actually, specifying package versions in the requirements.txt is a good practice that prevents this type of problems.

Comment

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: