During this session, I will share how with celery we can build better infrastructure designs for our Websites. I will emphasize more on the part’s where celery uses the full capabalities of Python’s Concurrency. I will share live demos and talk about concurrency in depth.
During this session i am going to talk on what is celery and how with the use of it we can build better infrastructure designs for our Websites. How the load on the Web frameworks can be reduced and make the whole flow and process more fast & efficient ultimately improving the User Experience. The main purpose of webframeworks like Django, flask etc is to provide tools and methods for efficiently handling requests and responses between users, databases, and other services. Your web framework is not meant to handle very high CPU intensive tasks and developers should know how to isolate these process. These tasks can be like scraping thousands of data from a website in the background, Running ML models in the background or sending 5000+ mails etc and for doing all these tasks celery comes into the play. Celery is a python framework which has task queue with focus on real-time processing, while also supporting task scheduling. Celery is used to handle heavy tasks in the background making the load less on web frameworks making it blazing fast. Celery requires a “message transport” to send and receive messages which can be redis, rabbitmq etc. I will use Redis in the live demo. Now celery handles the task asynchronously and has a lot of features to use concurrency to it’s full potential. In the talk i will discuss all the concepts about python’s concurrency like multithreading, multiprocessing, prefork, green threads, eventlet , gevent and more. I will show how celery uses CPU cores and threads to do the task as celery has a support for a lot of configurations in it and can be used accordingly to the demands. So we will also have discussions over threads, cores and process too! In the talk i will show live demos so audience can really understand it.
I’m a Python and Django Developer! I am a Django Software Foundation Member. I am also a DjangoNaut, Part of Team Saturn under Djangonaut Space Program. I love contributing to Open Source and most of the time i contribute to Django. I am a core member at DjangoIndia community, where we are trying to grow Django in India. I also maintain a few Open Source projects. I have a deep interest for System designing and Computer Networks.
I love to do some webscraping on the side and i also do devops!