taim

How We Built Taim: Process and Lessons Learned

Step 1: Working on architecture

When it comes to building a mobile application that relies on cutting-edge AI capabilities, the choice of technologies is paramount. Here's how we approached the architecture of Taim.

Multi-platform availability

Taim is not just another mobile application — it's a comprehensive solution that consists of:

• Mobile app for iOS: Designed to leverage Apple's ecosystem, providing a seamless experience for iPhone users.
• Mobile app for Android: Engineered to be fast, responsive, and intuitive across a variety of Android devices.
• Backend app: Serves as the bridge that facilitates communication with language learning models (LLM).
  ‍

Choosing Flutter for mobile frontend

We chose Flutter for both the iOS and Android mobile apps. This decision allowed our development team to be more agile, focusing on feature-rich development and rapid delivery. Flutter has been an instrumental choice right from the prototyping phase through development. The framework enabled us to maintain a smaller, more focused team without compromising on quality or speed.

Using Fastify for backend

Our backend application is crafted using Node.js with Fastify as the web framework. Fastify is designed for efficiency and a low learning curve, enabling us to build a robust backend that can handle the complexities of interacting with AI tools while being easy to maintain.

The LangChain and Helicone Combo for LLM

Interfacing with Language Learning Models (LLM) like OpenAI requires specialized tools. We leveraged two primary frameworks for this:

• LangChain: A comprehensive toolset that has become a standard in developing applications based on LLMs. It streamlines the development process, allowing us to focus on user experience and core functionality.
• Helicone: This is our go-to monitoring tool for LLM. It lets us keep tabs on costs, monitor prompts, and even implement caching for repeated articles. This real-time insight is crucial for maintaining the efficiency and cost-effectiveness of Taim.

The blend of these technologies ensures that Taim is not just another app but a well-thought-out solution designed to improve the way we consume information.

Step 2: Navigating the unpredictable: building products with AI

Incorporating AI into mobile applications brings a new layer of complexity, especially when dealing with non-deterministic technologies like language learning models (LLMs). Non-determinism, in the context of LLMs, means that the model can produce different outputs even when given the same input. This adds an extra layer of unpredictability that developers need to account for. While the possibilities are vast, so are the responsibilities.

Step 3: Moderating content for ethical compliance

AI, especially when interacting with external, user-generated content, can sometimes produce unexpected or inappropriate outputs. To mitigate this risk, Taim incorporates a text moderation step before summarization. We utilize a dedicated endpoint specifically designed to flag any content that violates ethical guidelines, such as hate speech or other inappropriate content. This ensures that the summaries generated align with societal norms and ethical considerations. We recommend starting with OpenAI's guidelines, which lays out comprehensive safety practices.

Step 4: The art of crafting prompts

Prompts in AI-driven applications are far from typical code fragments — they’re the questions that tell AI what you want from it. A well-crafted prompt can yield highly usable and accurate outputs, while a poorly-designed prompt can produce irrelevant or incorrect information. Throughout the development process, we experimented with dozens of different prompts for Taim's summarization feature.

To fine-tune this aspect, we developed an internal tool using Retool, which let us rate the quality of generated prompts and their respective summaries. Each team member could provide their subjective feedback, which was then compiled to identify which prompts resulted in the most accurate and useful summaries. This iterative, human-in-the-loop approach was invaluable for refining Taim’s AI components.

‍
Which Methods We Didn't Use and Why?
‍

In the realm of software development, striking a balance between quality and cost is crucial, especially for products aiming to scale. It's also important not to jump the gun by introducing complex optimizations prematurely. With Taim, we started with a prototype-focused approach, aiming to build an MVP that addresses the core needs of our users without unnecessary complexities. Below are some of the technologies and approaches we considered but ultimately decided to set aside.

Embeddings

The core idea of Taim is to succinctly summarize YouTube videos and articles. Even the longest articles and video subtitles are usually short enough to fit within the OpenAI context window. If not, Taim can easily manage it using a map-reduce algorithm to process larger chunks of text.

We considered adding embeddings as a feature to our app, especially because they could enable functionalities like search and article similarity. We decided to hold off on embeddings for several reasons:

• Most articles and YouTube videos can be fully summarized without text chunking.
• We want to prioritize user privacy and minimize data storage.
• Articles and their summaries are often read and used only once, making the need for search or similarity features less immediate.

Caching

We decided against using specialized caching techniques like GPTCache for cost optimization. Instead, we opted for the basic cache available through the Helicone platform. Implementing more advanced caching methods like GPTCache would necessitate the use of embeddings, which we chose to avoid for the reasons mentioned above.

GPT-4

While GPT-4 is the latest and greatest, Taim currently operates on GPT-3.5. We found that for the purpose of text summarization, GPT-3.5 is adequate. Sometimes, crafting a better prompt with GPT-3.5 can yield results just as good as those from GPT-4 but at a lower cost. Additionally, GPT-4 has stringent limitations, especially when it comes to tokens per minute. We are, however, in the process of allowing users to switch between GPT-3.5 and GPT-4 in Taim's settings.

Translating content to English

When using LLMs like OpenAI, you pay for the number of tokens processed. The number of tokens varies by language. For example, English typically uses fewer tokens compared to languages like German. Although translating content to English can be cost-effective, especially for recurrent tasks, we chose not to do it in Taim for several reasons:

• The summaries are usually generated only once.
• The cost of translating text would outweigh the token savings.
• Double translation (from the source language to English and back) can introduce inaccuracies.

By being judicious in our tech choices, we've made Taim not just powerful, but also efficient and user-friendly. Future iterations might include some of these omitted features as the app's needs evolve.

‍