Introduction
Let's face it, the clock speed of our CPU doesn't really affect the performance of simple programs highly in the modern age. Things like concurrency, multiprocessing or multithreading are the leading factors affecting the performance of those programs. Luckily with most of the current programming languages, it's really easy to implement those features. In this article, we're going to focus more on asynchronous(concurrency) programming, and more specifically the overlooked part in asynchronous programming; file handling.
Click here to learn more about multiprocessing or multithreading
What is Asynchronous Programming?
Asynchronous code just means that the language has a way to tell the computer/program that at some point in the code, it will have to wait for something else to finish somewhere else. Let's say that something else is called "slow-file". So, during that time, the computer can go and do some other work, while "slow-file" finishes.
In async, every function has a protocol of saying "I will take some time doing this please continue running other things'"
We can think of asynchronous programming in terms of hotels and waiters. Waiters come to you and give you a menu then they go away. They periodically come to you to check if you have completed picking the food, but while you pick the food, they go to other tables and take other orders.
In the same way, your async program also doesn't stop doing other tasks while waiting for a function to complete, it jumps back and forth from one task to another while waiting for the tasks to complete.
Why would you want to use async code?
One of the main uses of asynchronous codes is while I/O (hence the name asycnio). It takes around 126.95 seconds to retrieve data 500 times from an API in sync code. Whereas it takes 0.4880 seconds to retrieve data 500 times from an API in async code (For the same API). It's even faster and more efficient than threading and multiprocessing in the case of I/O because spawning threads and processes takes more time and resources than adding another coroutine in the event loop.
Click here for further reading.
Async Code Examples
Awaiting a function
await some_function()
NOTE: You can only await inside of async functions
Creating an Async function
async def some_function():
...
Running async function in root python
import asyncio
async def some_function():
...
if __name__ == '__main__':
asyncio.run(some_function())
Key Points
- Even though
multiprocessing, multithreading and concurrency may seem similar, they are different. You can check out the link below to learn more.
- async functions are also commonly referred to as
coroutines.
- asynchronous programming is also known as
concurrency.
Click here for further reading.
Credits: Nirjal Paudel, Mahesh C. Regmi
What is File Handling?
Simply put, file handling means reading and writing in the file through code (or specifically python in our case.) Through python, we can operate on the files. The concept of file handling is slightly different over various programming languages. In python's case, we can handle file handling with the help of the built-in function open().
We can open files in python in various modes such as:
- Read (r)
- Write (w)
- Append (a)
NOTE: There are a lot of other modes, you can learn about them here.
File Handling Code Examples
Assume we have an empty file 'random.txt':
Writing Content
with open('random.txt', 'w') as f:
for number in range(1, 4):
f.write(f"hello {number}\n")
If you check 'random.txt' you'll see these contents there:
hello 1
hello 2
hello 3
Reading content
with open('random.txt', 'r') as file:
content = file.read()
print(content)
OUTPUT:
hello 1
hello 2
hello 3
Click here for further reading.
You may be confused by the use of with while opening the file. with shows the usage of context manager in python.
What are Context Managers?
Usage of resources like file operations or database connections is very common in programming. But those resources are limited in supply. If we don't release these resources after using them, it may lead to resource leaking. This is the perfect use case for context managers, which automatically set up and teardown resources.
Click here for further reading.
Asynchronous file handing in python
The default file handling in python is synchronous hence blocking. It stops the execution of the whole program while doing any operations related to the file. The bigger the file, the more resource and time intensive it is, which may severely degrade the performance of the whole program. This is where asynchronous file handling comes into play.
We're going to use asyncio as the asynchronous runtime provider in python, and aiofiles python module for asynchronous file handling.
NOTE: You need to install aiofiles using pip install aiofiles for running the code examples below.
Using aiofiles
Asynchronously reading file contents
import asyncio
import aiofiles
async def main():
async with aiofiles.open('random.txt') as f:
contents = await f.read()
print(contents)
if __name__ == '__main__':
asyncio.run(main())
NOTE: You might have noticed that I haven't passed r as the mode. That is because r (read-mode) is the default mode, so we don't have to explicitly pass it every time.
It's going to have the same output as the file handling example above
Asynchronously iterating through lines of the file
async with aiofiles.open('random.txt') as f:
async for line in f:
print(f"New Line: '{line}'", sep=', ')
NOTE: I have removed the boilerplate code of imports, defining the main function and running of the function, if you want to run the code examples, you'll have to add that yourself.
OUTPUT:
New Line: 'hello 1', New Line: 'hello 2', New Line: 'hello 3'
Asynchronously writing to multiple files
import asyncio
import aiofiles
async def write_to_file(filename: str, content: str) -> None:
async with aiofiles.open(filename, "w+") as f:
await f.write(content)
async def main():
tasks = []
for number in range(100):
tasks.append(
asyncio.ensure_future(
write_to_file(f"{number}.txt", f"File Number: {number}")
)
)
await asyncio.gather(*tasks)
if __name__ == "__main__":
asyncio.run(main())
What is asyncio.ensure_future?
The function ensure_future lets us execute a coroutine (async function) in the background, without explicitly waiting for it to finish. If we need, we can wait for it later or poll for the result. In other words, this is a way of executing code in asyncio without await. Click here to learn more.
What is asyncio.gather?
The function gather lets you fire off a bunch of coroutines simultaneously. In other words, it returns the results of each coroutine (async function) passed The return value is a list of responses from each coroutine. Click here to learn more.
Task for the reader
- The reader can write the synchronous version of the above code and find out how the time difference between the 2 versions of code.
- The reader can write a program to read those 100 files you just created and keep the contents of every file in a list.
Resources for Learning More
Conclusion
File handling is often overlooked while doing asynchronous programming which leads to unintended slow ups as file handling is very resource and time-intensive for big files. Using aiofiles for file handling drastically decreases the execution time of the program due to its asynchronous nature. It does not have to block the whole program while reading a file, hence other files can be read/ write and modified in the meantime, so it's a lot faster than the default way of file handling in python.
