Archive

Pair Programming with Google's PaLM API.

Google Generative AI Studio: https://developers.generativeai.google/

What we need:

• API Key from above site
• Generative AI Libraries
  • node.js, swift or python
• Python programming skills (https://learnpython.org)

"Larger the animal, larger the model" - pip install -q google.generativeai

import google.generativeai as palm
palm.configure(api_key="....")
for m in palm.list_models():
	print(f"{m.name}")
	print(f"{m.description}")
	print(f"{m.supported_generation_methods}\n")

Course notes from Building Multimodal Search and RAG about Multimodal Learning.

• Instructor: Sebastian Witalec, head of developer relations at Weaviate,
• human's learn multi-modally
• human's foundational knowledge is built using multi-modal interaction with the world and not via language.
• multi-modal embedding models produce a joint embedding space that understand all your modalities (e.g.: text, video, audio, picture).
• Training:
  • start with specialist models
  • unify these models
  • the vectors that they generate are going to be similar
  • Models are unified using a process called contrastive representative learning.
    • CRL is a general purpose process that can be used to train any embedding model.
    • create one unified vector space representation for multiple modalities
      • by giving positive and negative examples, we train the models to pull closer vectors together for similar examples. and push away dissimilar examples
      • Anchor point
      • The pushing pulling process is achieved with the contrastive loss function.
        • encode the anchor and the (+ve and -ve ) examples into vector embeddings.
        • calculate the distance between the anchor and the examples.
        • minimize the distance between the positive example and maximize the negative example during the training process.
        • in multimodal learning, the examples can be from different modalities.
        • Minimize the loss... (Girdhar et al. 2023)

   

[via](https://learn.deeplearning.ai/courses/building-multimodal-search-and-rag/lesson/2/overview-of-multimodality)

References

• Girdhar, El-Nouby, Liu, Singh, Alwala, Joulin, Misra (2023). Imagebind: One Embedding Space to Bind Them All.