0

I'm developing a Forex trading application using Node.js, where I need to handle real-time updates for around 600 currency pairs. The data comes from a FIX protocol engine and needs to be updated in both Redis (for real-time updates to clients) and a MySql database (for persistence).

Problem: The server experiences high CPU and memory usage, leading to crashes. The main issue seems to be the frequent and numerous database updates.

const updateSymbolData = async (message) => {
  const entries = message.Body.NoMDEntries;
  const symbol = message.Body.Symbol.replace(/\.ecn|\.O|\.N/g, "");

  let bestBid = { price: 0, size: 0 };
  let bestAsk = { price: Infinity, size: 0 };

  entries.forEach((entry) => {
    if (entry.MDEntryType === "Bid" && entry.MDEntryPx > bestBid.price) {
      bestBid = {
        price: entry.MDEntryPx,
        size: entry.MDEntrySize,
      };
    } else if (entry.MDEntryType === "Offer" && entry.MDEntryPx < bestAsk.price) {
      bestAsk = {
        price: entry.MDEntryPx,
        size: entry.MDEntrySize,
     };
   }
 });

if (bestBid.price !== 0 && bestAsk.price !== Infinity) {
   await redisClient.set(`marketData:${symbol}`, JSON.stringify({ bestBid, bestAsk }));

  // Database update
  const query = `
    UPDATE forex_symbols
    SET best_bid = $1, best_ask = $2
    WHERE symbol = $3
  `;
  await queryAsync(query, [bestBid.price, bestAsk.price, symbol]);
 }
};

Questions: How can I optimize database updates to handle high-frequency updates efficiently? Is there a better way to manage real-time updates to minimize CPU and memory usage? What are best practices for synchronizing real-time data with a persistent database in such high-frequency scenarios?

Environment:

Node.js Redis MySql FIX Protocol for real-time data Any suggestions or best practices would be greatly appreciated!

Current Setup:

  1. Receiving Data: Using a FIX parser to receive real-time data
  2. Publishing to Redis: Publishing the updates to Redis channels for real-time client updates.
  3. Database Updates: Updating the MySql database with the latest bid and ask prices for each symbol.

0