Accelerating Video Uploads and Streaming with Express, Multer, and MongoDB in Docker

Howdy🙋. In today's digital age, video content is booming, and the need to efficiently store and stream video files has become crucial. Traditional methods of handling large video files can be cumbersome and slow. This blog will guide you through creating an efficient video upload and streaming API using Express, Multer, and MongoDB, all running inside Docker. We’ll focus on loading data in chunks to enhance speed and performance.

Why Load Data in Chunks?

Loading data in chunks is essential when dealing with large files, such as videos. It allows you to:

Reduce Memory Usage: Instead of loading an entire file into memory, you can process it in smaller, more manageable pieces.
Improve Upload and Download Speeds: By handling chunks, you can start processing parts of the file while still receiving it, leading to faster operations.
Enhance User Experience: Users can start watching parts of the video while the rest is still being downloaded.

How GridFS Facilitates Chunked Data Management

GridFS, MongoDB's specification for storing and retrieving large files, is designed to handle large files by splitting them into smaller, more manageable chunks. Learn more about Mongodb gridfs here.

Let's dive into the step-by-step process of setting up our API.

Step 1: Setting Up MongoDB in Docker

First, ensure you have Docker installed on your machine. We will create a Docker container for MongoDB via docker-compose file and also create a service for our express app.

version: "3.8"

services:
  mongodb:
    image: mongo:4.4
    container_name: mongodb
    ports:
      - "27017:27017"
    volumes:
      - mongo-data:/data/db

  app:
    build: .
    container_name: express-app
    ports:
      - "5080:5080"
    environment:
      - MONGO_URI=mongodb://mongodb:27017/f50
      - PORT=5080
    depends_on:
      - mongodb

volumes:
  mongo-data:

This sets up a MongoDB instance running on port 27017.

Step 2: Initialize Your Express Application

Next, create a new Node.js project and install the necessary dependencies:

npm init -y
npm install express mongoose multer multer-gridfs-storage mongodb

Step 3: Create the Express Server

Create a file named server.js and set up your Express server with routes for uploading and streaming videos.

const express = require("express");
const mongoose = require("mongoose");
const multer = require("multer");
const { GridFsStorage } = require("multer-gridfs-storage");
const { GridFSBucket } = require("mongodb");

// Allow cross-origin requests
const app = express();
const port = process.env.PORT || 5080;

app.use((req, res, next) => {
  res.header("Access-Control-Allow-Origin", "*");
  res.header("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");
  next();
});

// Connect to MongoDB
mongoose.connect(process.env.MONGO_URI, { useNewUrlParser: true, useUnifiedTopology: true });

// Initialize GridFS storage engine
const storage = new GridFsStorage({
  url: process.env.MONGO_URI,
  file: (req, file) => {
    return {
      filename: file.originalname,
      bucketName: "uploads",
    };
  },
});

// Initialize multer middleware
const upload = multer({ storage });

// Home route
app.get("/storage", (req, res) => {
  res.send("Welcome to the video storage API");
});

// Route for uploading video
app.post("/storage/api/v1/upload", upload.single("video"), (req, res) => {
  console.log("File uploaded successfully");
  res.send("File uploaded successfully");
});

app.get("/storage/api/v1/video/:filename", async (req, res) => {
  const filename = req.params.filename;

  // Get native MongoDB database object
  const db = mongoose.connection.getClient().db();

  // Create GridFSBucket instance
  const bucket = new GridFSBucket(db, {
    bucketName: "uploads",
  });

  try {
    const filesCollection = db.collection("uploads.files");
    const file = await filesCollection.findOne({ filename: filename });

    if (!file) {
      return res.status(404).send("File not found");
    }

    // Open download stream
    const downloadStream = bucket.openDownloadStream(file._id);

    res.set("Content-Type", "video/mp4");

    // Pipe download stream to response
    downloadStream.pipe(res);
  } catch (error) {
    console.error("Error retrieving file:", error);
    res.status(500).send("Error retrieving file");
  }
});

app.listen(port, () => {
  console.log(`Server is running on http://localhost:${port}`);
});

Step 4: Setting up Dockerfile for our express app

# Use the official Node.js image as a base image
FROM node:18-alpine

# Set the working directory inside the container
WORKDIR /app

# Copy package.json and package-lock.json to the working directory
COPY package*.json ./

# Install the dependencies
RUN npm install

# Copy the rest of the application code to the working directory
COPY . .

# Expose the port the app runs on
EXPOSE 5080

# Start the application
CMD [ "npm", "start" ]

Step 5: Running the Application

docker compose --build -d

Your application should now be running at http://localhost:5080. You can upload videos via the /storage/api/v1/upload route and stream them through /storage/api/v1/video/:filename.

Conclusion

By loading video data in chunks, this setup ensures efficient memory usage and faster upload/download speeds, significantly enhancing the user experience. This blog should help you get started with building a scalable video storage and streaming solution using Express, Multer, and MongoDB in Docker. Happy coding!