Building Senseii: The Backend - Part 1

Hey! Prateek here, this is a continuation of the "Building Senseii" series, in this one, we are going to talk about how we are implementing our backend, which is big, so it will be split into multiple parts. let's get started.

Architecture

Our backend API is an express, Node JS server, that is built using TypeScript. We are using MongoDB as our database and OpenAI's Assistant API is the backbone of this Product.

The overall architecture is pretty much straightforward, Senseii is an environment, where multiple agents talk to each other to assist the user with their needs. Each agent adding a new feature to the environment.

For v1 of the product, we want to focus on Fitness and Health, so we sat down and thought, what types of agents do we need for that. After a good amount of research through online forums of "how fitness works" and platforms like Cult Fitness, we realized, two major needs are, "Nutrition" and "Workout" Assistant. Where the Nutrition assistant is an expert in Nutritional Sciences, and Workout assistant is an expert in human body, movement and exercises. Compare them to your certified nutritionists and Gym trainers in real life.

We are using OpenAI's assistant API, so every Agent is an assistant, with additional functionalities. We are leveraging function calling, where each agent has its own functions, that other Agents can call depending on what the user needs.

So, currently the system has three Assistants, namely:

• Core: The only interface between the user and the Application.

• Nutrition: Expert in Nutrition sciences.

• Workout: Expert in human body, movements, workout etc.

Let's take an example now, user wants to lose a certain amount of weight, they tell it to the core assistant, which in turn, asks additional information, so that the system has some information about the user. The collected information looks something like this:

{
  "basicInformation": {
    "age": 22,
    "weight": 106,
    "height": 180,
    "gender": "male"
  },
  "lifeStyle": {
    "dailyRoutine": "moderate",
    "exerciseRoutine": {
      "exerciseType": "strength",
      "frequency": "weekly"
    }
  },
  "dietPreferences": {
    "preference": "non-vegetarian",
    "allergies": [],
    "intolerances": [],
    "dislikedFood": [],
    "favouriteFood": []
  },
  "healthGoals": {
    "weightGoal": "loss",
    "specificNutritionGoal": "weight loss",
    "medicalConditions": []
  },
  "eatingHabits": {
    "mealsPerDay": 3,
    "mealComplexity": "simple",
    "cookingTime": "less than 30 minutes"
  },
  "constraints": {
    "financial": {
      "budget": 5000,
      "budgetType": "monthly"
    },
    "geographical": {
      "location": "India"
    }
  }
}

I used a simple prompt like "Collect all this information from the user, and format it in the above JSON structure", as a result of which, core assistant engages in communication and retrieves all the information relevant to the above structure, and fills in the details like location, health goals, etc.

Next step is, creating a nutrition and workout plan for the user, which they are supposed to follow. Our nutrition assistant has functions like create-nutrition-plan, update-nutrition-plan attached to it, similarly for the fitness assistant. and these functions are callable by the core agent, because mood of the core agent, is the mood of the entire application. He is the real 'Sensei'!

Core assistant passes the collected user information to the Nutrition and Workout assistants, which in turn respond with a diet and a workout plan, which user can keep altering until they finally agree upon. Then the entire process of Progress Tracking Starts

Let's keep the process of "progress tracking" for the next article.

Blockers

We want to make these Assistants experts in their fields, and that is not an easy task to do. There are multiple strategies, and they all need to be tested one by one, and not to forget, they all have their own costs.

Currently, we are testing out the MedPrompt strategy, suggested by Microsoft in their paper here Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine which needs us to have a good sample data, which hypothetically be generated by models on the run, but still, we want some good samples to shape the application flow as well.

A call for help to all the nutritionists in the world, who are willing to collaborate with us, we need your help to fully understand the process, you follow. Help us make health and fitness more accessible!

Till then, see you next time.