Our client, a MarTech startup, envisioned an app specifically tailored to the needs of today's content creators. Their users – influencers, bloggers, and small businesses, would use the app to generate high-quality textual content for social media, edit and schedule their posts seamlessly, streamlining their workflow and freeing them from time-consuming tasks. This allowed them to focus on what truly matters: expressing their creativity and engaging their audiences.

When the company first came to us, they had an idea for the project and looked for someone that could bring it to life.Their vision was clear: upload a photo, describe it via text or voice, and the app generates optimized text with hashtags, ready to be shared. Alternatively, an app would identify the photo's content and create a text without any additional input from the user. Users could connect their social media accounts with the app to schedule posts, publish instantly, or even create multiple posts in one go.

It was February 2023, a pivotal moment for the industry, as OpenAI had just announced the public release of ChatGPT Plus and Whisper model APIs. We had the chance to  experiment with the technology already, but didn’t have a project where we could fully implement it. This project came just at the right time, so we were happy to start working. By this time, few companies had fully integrated generative AI into their workflows, allowing us to become one of the first companies to experiment with this technology.

Our collaboration with the client began with an idea and a shared sense of excitement for the possibilities of generative AI. We first started with the initial discovery phase. Taking us around a month, it, involved:

• Understanding needs: We delved into the client's goals, target audience, and desired functionality.
• Planning and specifications: We charted a roadmap, outlined technical specifications, and drafted initial design concepts.
• Trying out the technology: With generative AI being a relatively new frontier, we spent time learning and experimenting to understand its capabilities and limitations.

Creating an AI-based app presented our team with several challenges: 

1. Decoding User Intent & Crafting Compelling Content 

The most important part was assuring that the core functionality of the app was stable. It had to accurately decipher user intent and adapt the texts to various social media platforms while adhering to their guidelines. It also needed to adjust the tone of voice to sound natural and resonate with the intended audience. To achieve this, we extensively experimented with prompt engineering, creating advanced prompts that would consistently provide stable and accurate results. 

2. Merging Technologies 

Integrating speech-to-text and image recognition models presented another challenge. Speech-to-text models convert spoken words into text, while image recognition models interpret visual information. The app needed to seamlessly translate between these different data formats, ensuring consistent representation and understanding. It also needed to process both inputs simultaneously and combine them into a cohesive post. This meant ensuring all models completed their tasks promptly and the app assembled the final output in the correct order, creating a natural and uninterrupted user experience.

3. Creating Recognizable Interface 

We also had the goal to maintain a user-friendly interface that would work and feel as natural as the social media apps the users were used to, providing users with a comfortable starting point. At the same time, we didn’t want to create a carbon copy. The interface needed to seamlessly integrate with the app's own design identity, fostering a unique visual experience. 

Additionally, ensuring smooth interaction with various social media platforms through integrated APIs and minimizing buffering times were crucial for a responsive and enjoyable user experience. 

4. Security and Privacy:

Security and privacy were cornerstones of the project. We envisioned a robust platform capable of handling a large user base with unwavering stability, even under significant load. Data protection was equally important. Implementing robust measures to safeguard user privacy and adhering to strict GDPR regulations became a priority, since we would have access to sensitive data. 

Following a meticulously planned and collaborative development process spanning six months, the first complete version of the app launched in August 2023. From then on, we continued working on the app, providing post-launch support.