Redis In Action: Twitter Dashboard Simulation

Table Of Content

• Redis In Action: Twitter Dashboard Simulation
  • Table Of Content
  • Introduction:
  • Features:
    • Creation:
    • Read:
    • Update:
    • Delete:
    • Console:
    • References

Introduction:

The Project consists of a Tweeter Dashboard Simulation, that is responsible for simulating real world operation. Redis is used as the In-Memory Database, alongside an RDBMS, Postgres. Postgres acts like a primary central database, assuring ACID properties, while REDIS plays the role of storing output of queries. In this way, Redis DB acts like a cache system.

The project also uses a real world Tweets dataset, having 100,000 tweets, this dataset is used to simulate High Read/Write operations.

The Frontend is built using React with Typescript, while Flask (Python) is used in the backend, using official databases supported libraries.

Features:

All the database operations for Redis/Postgres are tracked in milliseconds and displayed on the console visible on the front-end.

Creation:

• A Largedataset consisting of 100K Tweets is used as the primary source for adding tweets to the database. This dataset is available in CSV file.

• Frontend provides both the options to insert CSV tweets by accepting a range of ROWS and an individual tweets with all required parameters.

• In one batch all 100k tweets can be inserted, however this operation is time consuming, consuming about 10-15 mins. This is not due to the database but IO overhead caused by reading from CSV, converting to Python Object then inserting to DB/Cache.

• (Response)

• Inserting a single tweet is trivially fast

• (Response)

Read:

• User is provided the option to fetch All Tweets or Specific User’s Tweets (By User-Id).

• A Choice for source is provided to the user. Either Postgres or Redis. Selecting Postgres would invoke Postgres to retrieve tweets from its db. While, selecting Redis would return from cache, which is always faster than fetching from DB, by almost 80% is some cases.

• Along with tweets, it is possible to compute statistics about the data present on the RDBMS, which is an expensive operation, the result is stored in Redis.

• Statistics shows are: Retweet Count, Most Retweets, Total Reach, Total Number of Likes.

• (Statistics)

Update:

• All Tweets are editable. Only the Tweet content can be updated.
• The update is triggered over the Postgres DB and does not affect the cache.

Delete:

• Tweets can be deleted, individually.
• Deleted Tweets are reflected on the Primary database, Postgres. However they’re still present in the cache
• Database drop is also supported, in this all the tweets are deleted from both, Postgres and Redis. Along with all the additional keys in redis.
• This would reset the state of the application

Console:

• Displays the Logs of the operations.
• It includes time consumed for database read/write operations.
• It is available for both the redis and postgres.
• This provides a neat measure to benchmark the Databases by performance and speed

---

References

• Redis Documentation
• Redis-Py - Python Redis Client
• Postgres Documentation
• SQL Alchemy Tutorial - Python Central
• SQL Alchemy
• Flask
• Flask - SQLAlchemy