Simulating 500 million years of evolution with a language model
Revolutionary protein LLM that combines sequence, structure, function in one generative model. Combining function and structure into this same model is a similar acheivement to the latest LLM multimodal models that incorporate text and images into a single multimodal model.
Works remarkably well at all scales with the 98B model acheiving truly impressive results purely from sequence.
Performs less well than alphafold2. alphafold2 performs significantly worse than alphafold3. In addition recent work 10.1038/s41592-023-02087-4 shows that AlphaFold predictions are valuable hypotheses and accelerate but do not replace experimental structure determination.
After being laid off from Facebook ex-meta meta-axed they started a new company EvolutionaryScale funding by Lux Capital, Nventures (Nvidia) and AwS.
ESM3 latest model from the Protein AI team formerly Facebook at now at EvolutionaryScale.
Allows for incorporation of sequence, structure and function in a masked transformer.
Three scales of model: 1.4 billion, 7 billion, and 98 billion parameters.
1.4 Billion parameter model released on github https://github.com/evolutionaryscale/esm
Left Facebook and becoming Evolutionary Scale. $142 million in seed funding cen.acs.org
ESMFold performed significantly worse than Alphafold2 and other competitors in CASP15. https://predictioncenter.org/casp15/zscores_final.cgi
Alphafold3 is now released. Results in paper show Alphafold2 numbers. Alphafold3 is substantially more accurate than Alphafold2. 10.1038/s41586-024-07487-w
CASP16 is underway right now, finishing soon. Interesting time to release this paper as a preprint and then to bioarxiv.
Experiment using GFP as a self-reporter is creative and elegent, but unorthodox. Some of their designed fluorophores take days to develop.
Total cost of training model is estimated at approximatly $12M. Total seed funding is $140M. Estimate for total required is $268 ex-meta.
The company aims to establish revenue streams through partnerships, usage fees, and revenue sharing agreements, with potential collaborations with pharmaceutical companies and researchers to incorporate ESM3 into their workflows.
"an effort that may span a decade" ex-meta
Their experiment basically ensures that they find a fairly distant relative of GFP if it exists, and it does exist and is called tagRFP, an existing Red Fluorescent Protein from another organism that. tagRFP has a similar fold to esmGFP but makes a different flurophore.
They then did a second experiment to improve the functioning of this GFP and recovered the function.
https://www.sciencedirect.com/science/article/pii/S0959440X23000684
OmegaFold [43] and ESMfold [44] are two implementations that seem similar to AlphaFold but without using the MSA. However, whether the predictions using these models use a single sequence can be questioned. The performance is significantly worse for (orphan) proteins that do not have many homologs in the sequence databases, i.e. the language models appear to memorise the MSA. ESMfold is computationally efficient and has been used to predict the structure of all proteins from an extensive meta-genomics database [45]. At CASP15, these methods performed significantly worse than AlphaFold.
retrieved from: https://evolutionaryscale-public.s3.us-east-2.amazonaws.com/research/esm3.pdf
Preview 2024-06-25. Pending submission to bioRxiv. Copyright 2024 by the authors.
Alexander Rives - Evolutionary Scale Language Models
Alexander Rives - Evolutionary Scale Language Models
Great video by Alexander Rives, long and technical with lots of great information. Wonderful overview of the field of generative AI for proteins. Interesting aside in the questions at the end about the accessiblity of other researchers in academia competeing with companies like Meta and Google with their vast resources, where he mentions that he doesn't know how much the models cost to train.
Protein Language Models
Really nice background on protein LLMS from Rosetta Commons.
In this video, language models are broken down in broad strokes and are reimagined for protein modeling, wherein protein sequence patterns are identified and then used to predict new sequences following the learnable parameters. The video explores IGLM as a case study, covering generative language models, controllable generation of sequences, infilled protein libraries, likelihood estimations.
Efficient Evolution of Human Antibodies From General Protein Language Models and Sequence Info Alone
Brian Hie, Stanford & Meta AI
Abstract: Natural evolution must explore a vast landscape of possible sequences for desirable yet rare mutations, suggesting that learning from natural evolutionary strategies could accelerate artificial evolution. Here, we report that deep learning algorithms known as protein language models can evolve human antibodies with high efficiency, despite providing the models with no information about the target antigen, binding specificity, or protein structure, and also requiring no additional task-specific finetuning or supervision. We performed language-model-guided affinity maturation of seven diverse antibodies, screening 20 or fewer variants of each antibody across only two rounds of evolution. Our evolutionary campaigns improved the binding affinities of four clinically relevant antibodies up to 7-fold and three unmatured antibodies up to 160-fold across diverse viral antigens, with many designs also demonstrating improved thermostability and viral neutralization activity. Notably, our algorithm requires only a single wildtype sequence and computes recommended amino acid changes in less than a second. Moreover, the same models that improve antibody binding also guide efficient evolution across diverse protein families and selection pressures, indicating that these results generalize to many natural settings. Contrary to prevailing notions of evolution as difficult and resource-intensive, our results suggest that when constrained to a narrow manifold of evolutionary plausibility, evolution can become much easier, which we refer to as the “efficient manifold hypothesis.”
Meta ESM-2 Fold - AI faster than Alphafold 2
Fairly standard AI pumping video, this time on ESMfold.
https://www.youtube.com/embed/aPCqzrscY4w
Another AI pumping video for ESM. Fairly good.
The State of Protein Structure Prediction and Friends
The State of Protein Structure Prediction and Friends Mohammed AlQuraish (Columbia)
AlphaFold2 revolutionized structural biology by accurately predicting protein structures from sequence. Its implementation however (i) lacks the code and data required to train models for new tasks, such as predicting alternate protein conformations or antibody structures, (ii) is unoptimized for commercially available computing hardware, making large- scale prediction campaigns impractical, and (iii) remains poorly understood with respect to how training data and regimen influence accuracy. Here we report OpenFold, an optimized and trainable version of AlphaFold2. We train OpenFold from scratch and demonstrate that it fully reproduces AlphaFold2's accuracy. By analyzing OpenFold training, we find new relationships between data size/diversity and prediction accuracy and gain insights into how OpenFold learns to fold proteins during its training process.
EvolutionaryScale, a new startup founded by former Meta researchers who specialized in AI language modeling for biology, has successfully raised $40 million in seed financing, led by Lux Capital. The company, valued at $200 million, aims to advance the development of biological large language models (LLMs), with significant investments from AI investors Nat Friedman and Daniel Gross.
The core innovation backing EvolutionaryScale lies in its founders' prior work at Meta AI, where they developed a transformer-based model for predicting protein structures, analogous to AI models like OpenAI's GPT-4. This model has already contributed to creating a database containing 700 million potential 3D protein structures, a resource that holds promise for accelerating drug discovery, environmental cleanup efforts, and the synthesis of industrial chemicals by providing insights into the molecular architecture of proteins and their interactions.
EvolutionaryScale's strategy hinges on the "scaling hypothesis" for AI in biology, betting that enlarging the model's dataset and complexity will lead to breakthroughs similar to those seen in natural language processing around 2018. They project significant investments in computing power, forecasting expenditures of up to $278 million in the third year, with the goal of expanding their model to incorporate a wide array of biological data sources beyond protein structures.
Despite the competition from established players like DeepMind's AlphaFold and emerging startups aiming to leverage LLMs for biomedical applications, EvolutionaryScale aspires to differentiate itself by developing a versatile and general-purpose AI model for biology. This ambition is underscored by lofty goals, including the design of programmable cells for cancer treatment and molecular machines for environmental restoration.
Given the founders' background, the project is receiving notable attention in both the biotech and AI fields. However, the company is clear about the challenges ahead, emphasizing a long-term vision that might require a decade to realize fully, particularly in terms of producing usable products and therapies. This initiative represents a significant and bold investment in the intersection of AI and biotech.
https://colab.research.google.com/github/evolutionaryscale/esm/blob/main/examples/generate.ipynb
Nice Google Colab to demonstrate ESM. It has nice visualizations of the proteins, although the proteins should probably be superimposed, not side by side. This is a TODO(sness) item.
https://github.com/evolutionaryscale/esm
They have a really nice way to run ESM using HuggingFace, a pip package for esm and some nice sample code. I got this running, pretty easy to get going. Unfortunately only the 1.4 billion parameter model of course.
https://ca.finance.yahoo.com/news/evolutionaryscale-launches-esm3-milestone-ai-100000341.html
https://www.reddit.com/r/singularity/comments/1dole9a/esm3simulating500millionyearsofevolution/
https://www.php.cn/faq/1796510087.html
https://axios.com/2024/06/25/ai-biotech-generative-model-protein-design
https://evolutionaryscale-public.s3.us-east-2.amazonaws.com/research/esm3.pdf
https://www.nature.com/articles/s41592-023-01790-6
https://analyticsindiamag.com/protein-wars-its-esmfold-vs-alphafold/
https://salvatore-raieli.medium.com/metas-esmfold-the-rival-of-alpahfold2-2223b67f6021
GFP search term yields lots of ads with Google Search
We used esmfold in htgaa 2023. We did not use it in 2024 and just used Alphafold3. Alphafold3 is really awesome btw.
Here is a student that used it:
https://complex-bike-918.notion.site/Protein-Design-df2978ed760e4f368b0d236b40212b01
Aaron W Kollasch Adam Lerer Ada Shaw Alec Radford Alessandro Barbato Alethea Power Alexander Derry Alexander Rives Alex Nichol Alex Paino Alex Ray Amos Bairoch Andrew N Carr Andriy Kryshtafovych Ariel Herbert-Voss Bilal Piot Bob McGrew Brian Hie Brooke Chan Carolyn Kim Charlotta PI Schärfe Chengxin Zhang Chetan Mishra Chloe Hsu Chris Sander Christian J A Sigrist Christopher Hesse Clemens Winter Daniele Calandriello Daniel Guo Daniel Ritter Dan Jurafsky Dario Amodei Dario Behringer Daron M Standley Dave Cummings Davis Liang Debora S Marks Deniz Oktay Dinko Franceschi Douwe Kiela Edouard De Castro Edward J Hu Elizabeth Barnes Evan Morikawa Felipe Petroski Such Fotios Chantzis Frank J Poelwijk Gabriel Studer Girish Sastry Greg Brockman Gretchen Krueger Halil Akin Hansen Spinner Harri Edwards Heewoo Jun Heidy Khlaaf Henrique Ponde de Oliveira Pinto Igor Babuschkin Ilya Sutskever Irhum Shafkat Jacob Hilton James Melville Janani Durairaj Jan Leike Jared Kaplan Jason Liu Jerome Eberhardt Jerry Tworek Jie Tang John B Ingraham John Healy John Moult John Schulman Jonathan Deaton Jonathan Frazer Josef Perktold Josh Achiam Julian B Kinney Julian Salazar Jun Gong Jürgen Haas Katie Mayer Katrin Kirchhoff Kawin Ethayarajh Kazutaka Katoh Khaled Mostaguir Krzysztof Fidelis Leland McInnes Leo Gao Liam J. Bartie Lood van Niekerk Lorenzo Cerutti Lukasz Kaiser Lu Wang Maciej Antczak Mafalda Dias Marco Pagni Marius Wiggert Mark Chen Mark Rowland Marta Szachniuk Martino Bertoni Matthew Knight Matthias Plappert Maya Topf Michael Petrov Michael Springer Michal Valko Mikhail Pavlov Miles Brundage Mira Murati Mohammad Bavarian Mohammad Gheshlaghi Azar Nathan Rollins Neil Thomas Nicholas James Sofroniew Nicholas Joseph Nick Ryder Nicolas Hulo Nikita Smetanin Niklas Muennighoff Nikolas Tezak Ori Kabeli Pamela Mishkin Pascal Notin Patrick D. Hsu Peter L Freddolino Peter Welinder Petra S Langendijk-Genevaux Philippe Tillet Phillip Pham Phillip Wallis Qiming Yuan Rachael C Kretsch Rafal Gumienny Ramya Rangan Raul Puri Raul Santiago Molina Rémi Munos Rhiju Das Robert Verkuil Rohil Badkundri Rose Orenbuch Roshan Rao Ruben Weitzman Salvatore Candido Sam McCandlish Scott Gray Shantanu Jain Shean Wang Skipper Seabold Steffanie Paul Steven Roth Suchir Balaji Talley J Lambert Thomas A Hopf Thomas Hayes Toan Q Nguyen Tomasz Zok Tom Sercu Torsten Schwede Vedant Misra Vincent Quy Tran Virginie Bulliard Weizhu Chen William Hebgen Guss William Saunders Winnie Xu Wojciech Zaremba Xavier Robin Xi Zhang Yang Zhang Yarin Gal Yelong Shen Yousuf Khan Yuanzhi Li Yuri Burda Zeming Lin Zeyuan Allen-Zhu
Thomas Hayes 1 & Roshan Rao 1 & Halil Akin 1 & Nicholas James Sofroniew 1 & Deniz Oktay 1 & Zeming Lin 1 & Robert Verkuil 1 & Vincent Quy Tran 2 3 Jonath7an Deaton 1 Marius Wiggert 1 Rohil Badkundri 1 Irhum Shafkat 1 Jun Gong 1 Alexander Derry 1 Raul Santiago Molina 1 Neil Thomas 1 Yousuf Khan 4 Chetan Mishra 1 Carolyn Kim 1 Liam J. Bartie 2 Patrick D. Hsu 2 3 Tom Sercu 1 Salvatore Candido 1 Alexander Rives 1 †
1 EvolutionaryScale, PBC 2 Arc Institute 3 University of California, Berkeley 4 Work done during internship at EvolutionaryScale, PBC
†Correspondence to arives@evolutionaryscale.ai.
More than three billion years of evolution have produced an image of biology encoded into the space of natural proteins. Here we show that language models trained on tokens generated by evolution can act as evolutionary simulators to generate functional proteins that are far away from known proteins. We present ESM3, a frontier multimodal generative language model that reasons over the sequence, structure, and function of proteins. ESM3 can follow complex prompts combining its modalities and is highly responsive to biological alignment. We have prompted ESM3 to generate fluorescent proteins with a chain of thought. Among the generations that we synthesized, we found a bright fluorescent protein at far distance ( $58 \%$ identity) from known fluorescent proteins. Similarly distant natural fluorescent proteins are separated by over five hundred million years of evolution.
The proteins that exist today have developed into their present forms over the course of billions of years of natural evolution, passing through a vast evolutionary sieve. In parallel experiments conducted over geological time, nature creates random mutations and applies selection, filtering proteins by their myriad sequences, structures, and functions. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - Proteins have developed into their current forms over billions of years.
As a result, the patterns in the proteins we observe reflect the action of the deep hidden variables of the biology that have shaped their evolution across time. Gene sequencing surveys of Earth's natural diversity are cataloging the sequences $(1-3)$ and structures $(4,5)$ of proteins, containing billions of sequences and hundreds of millions of structures that illuminate patterns of variation across life. A consensus is building that underlying these sequences is a fundamental language of protein biology that can be understood using large language models (6-10). Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - The observed patterns in proteins are shaped by the underlying biology that has evolved over time.
A number of language models of protein sequences have now been developed and evaluated ( $9,11-14$ ). It has been found that the representations that emerge within language models reflect the biological structure and function of proteins $(6,15,16)$, and are learned without any supervision on those properties, improving with scale $(5,17,18)$. In artificial intelligence, scaling laws have been found that predict the growth in capabilities with increasing scale, describing a frontier in compute, parameters and data (19-21). Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - A number of language models of protein sequences have been developed and evaluated.
We present ESM3, a frontier multimodal generative model, that reasons over the sequences, structures, and functions of proteins. ESM3 is trained as a generative masked language model over discrete tokens for each modality. Structural reasoning is achieved by encoding three-dimensional atomic structure as discrete tokens rather than with the complex architecture and diffusion in three-dimensional space employed in recent predictive (22) and generative models $(14,23-25)$ of proteins. All-to-all modeling of discrete tokens is scalable, and allows ESM3 to be prompted with any combination of its modalities, enabling controllable generation of new proteins that respect combinations of prompts. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - ESM3 is a frontier multimodal generative model.
ESM3 at its largest scale was trained with $1.07 \times 10^{24}$ FLOPs on 2.78 billion proteins and 771 billion unique tokens, and has 98 billion parameters. Scaling ESM3 to this 98 billion parameter size results in improvements in the representation of sequence, structure, and function, as well as on generative evaluations. We find that ESM3 is highly responsive to prompts, and finds creative solutions to complex combinations of prompts, including solutions for which we can find no matching structure in nature. We find that models at all scales can be aligned to better follow prompts. Larger models are far more responsive to alignment, and show greater capability to solve the hardest prompts after alignment. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - ESM3 was trained with $1.07 \times 10^{24}$ FLOPs.
Please estimate the cost of training this modeL
AI: To estimate the cost of training the ESM3 model with 98 billion parameters, we will rely on information provided and extrapolate using approximate costs for training deep learning models of similar scale as of 2023.
Given the significant assumptions and variability in cloud pricing, discounts, and efficiencies, this estimate is very rough. The actual cost could be lower or higher based on these factors. Additionally, advancements in technology and cloud infrastructure could also influence training costs.
We report the generation of a new green fluorescent protein (GFP) with ESM3. Fluorescent proteins are responsible for the glowing colors of jellyfish and corals (26) and are important tools in modern biotechnology (27). They share an elegant structure: an eleven stranded beta barrel with a helix that threads its center, which scaffolds the formation of a light-emitting chromophore out of the protein's own atoms. This mechanism is unique in nature-no other protein spontaneously forms a fluorescent chromophore out of its own structure-suggesting that producing fluorescence is hard even for nature. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - A new green fluorescent protein (GFP) with ESM3 has been generated.
Our new protein, which we have named esmGFP, has $36 \%$ sequence identity to Aequorea victoria GFP, and $58 \%$ sequence identity to the most similar known fluorescent protein. Despite GFP's intense focus as a target for protein engineering over several decades, as far as we are aware, proteins this distant have only been found through the discovery of new GFPs in nature.
ME: Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - The new protein is named esmGFP.
Similar amounts of diversification among natural GFPs have occurred over predictable timescales. Understood in these terms, the generation of a new fluorescent protein at this distance from existing proteins appears to be equivalent to simulating over 500 million years of evolution.
ME: Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - Similar amounts of diversification have occurred among natural GFPs (Green Fluorescent Proteins).
ME: Please analyze this from the point of view of an reviewer expert in both LLMs and biotechnology.
AI: As an expert reviewer with a background in both large language models (LLMs) and biotechnology, the generation of a new green fluorescent protein (GFP) through the use of ESM3—presumably an advanced machine learning algorithm or platform—is both intriguing and a noteworthy advancement in the field. My analysis will cover the novelty of the approach, its technical validity, implications for the field, and potential areas for further exploration.
The development of esmGFP marks a significant achievement given its low sequence identity to Aequorea victoria GFP (36%) and yet a relatively higher identity to the closest known fluorescent protein (58%). The disparity in sequence identity with known GFPs highlights the innovative success of applying ESM3 in discovering or designing novel proteins, potentially opening new avenues for similar advancements using computational models. This approach indicates a paradigm shift from traditional experimental or evolutionary discovery processes to computational design and simulation, aligning with the growing integration of artificial intelligence (AI) in biotechnological advancements.
The claim of simulating over 500 million years of evolution is profound and necessitates a rigorous examination of the methodologies employed. The selection criteria, machine learning algorithms, datasets, and computational parameters integral to ESM3's function in generating esmGFP are critical for assessing the model's robustness and the novelty's reproducibility. Additionally, experimental validation of esmGFP's fluorescence, structural stability, and functionality compared to naturally occurring GFPs is essential to support the computational findings. This dual approach enhances the scientific credibility of AI-generated biological innovations.
The utility of GFPs in molecular and cellular biology as tags for gene expression, protein localization, and interaction studies has been transformative. An AI-driven leap in fluorescent protein engineering could expand the toolkit available to researchers, offering proteins with potentially novel properties (e.g., differing wavelengths of fluorescence, enhanced stability, or reduced toxicity). Such advancements could significantly impact molecular biology, diagnostic procedures, and therapeutic applications, pushing the boundaries of what is currently achievable.
Several areas warrant further exploration following the generation of esmGFP. Initially, understanding the structural and functional properties of esmGFP in comparison to naturally occurring counterparts would be invaluable. Investigating how esmGFP behaves in different environments and in vivo systems could highlight its potential applications and limitations. Furthermore, exploring the scalability of the ESM3-driven process for generating other proteins with desirable characteristics could revolutionize protein engineering.
Additionally, ethical considerations surrounding AI-generated biological materials must be addressed, including safety, patentability, and accessibility. As computational methods accelerate the pace of biological discoveries, establishing guidelines and regulatory frameworks will be imperative.
In summary, the creation of esmGFP through ESM3 represents a compelling intersection of computational science and biotechnology, showcasing the potential to accelerate evolutionary processes and innovate beyond natural constraints. While the technical and ethical aspects demand thorough examination, the overarching trajectory indicates a promising future for AI in advancing biotechnological research and applications. This work not only underscores the importance of interdisciplinary collaboration but also reinforces the transformative impact of computational tools in uncovering the mysteries of biology.
ME: Please consider how RFPs and how they are related to GFPs.
AI: Red fluorescent proteins (RFPs) are related to green fluorescent proteins (GFPs) in several fundamental ways, both being part of the broader family of fluorescent proteins (FPs) that are critical tools in molecular and cellular biology. The relationship between RFPs and GFPs, and their relevance to the generation of the new green fluorescent protein (esmGFP) using ESM3, offers further dimensions to the analysis from both the biotechnological and computational perspectives.
Structural Homology: GFPs and RFPs share a similar basic structure, which is the β-barrel structure wrapping around the chromophore. This conserved architecture facilitates the process where the protein absorbs light at one wavelength and emits it at another, creating fluorescence. While the chromophores within GFPs and RFPs differ in structure leading to the emission of green and red light, respectively, these differences arise from variations in the sequence that affect the post-translational modifications and the environment of the chromophore.
Evolutionary Link: GFPs and RFPs are evolutionarily related. They originated from a common ancestral protein, which has diverged through mutations and natural selection to produce proteins with different fluorescent properties. This evolutionary link is crucial when considering the diversification among natural FPs and the simulated evolution used in generating esmGFP, as it highlights the potential for discovering or designing FPs across the spectrum by mimicking or accelerating evolutionary processes through computational approaches.
Expanding the Fluorescent Protein Palette: The creation of esmGFP through ESM3 underscores the potential to extend these computational methodologies to engineer not just GFP variants but also RFPs and other FPs with desired properties, including altered fluorescence, improved stability, or enhanced brightness. By leveraging artificial intelligence and sophisticated algorithms, researchers can explore a vast space of genetic sequences that natural evolution might not have accessed, potentially creating a diverse palette of FPs for advanced imaging techniques.
Biotechnological and Research Applications: GFPs and RFPs are foundational in biotechnology for tagging proteins, tracking gene expression, visualizing cellular processes, and more. By designing novel GFPs (like esmGFP) and extending these efforts to RFPs and beyond, researchers can develop more sensitive and versatile tools for real-time imaging and diagnostic applications, enabling the study of biological processes with unprecedented resolution and spectral range.
Interdisciplinary Collaboration: The evolution from traditional cloning and harvesting of FPs from natural sources to the computational generation of esmGFP illuminates the importance of interdisciplinary collaboration. Combining computational biology, machine learning, genetic engineering, and biochemistry is essential for the next wave of discoveries in the fluorescent protein domain.
The interconnectedness between GFPs and RFPs, through structural, functional, and evolutionary parallels, enriches the significance of developing esmGFP using a computational approach like ESM3. This strategy not only presents a platform for the innovative generation of FPs but also paves the way for a broadened understanding of protein engineering, offering exciting possibilities for research and practical applications in biotechnology and beyond.
ESM3 reasons over the sequence, structure, and function of proteins. All three modalities are represented by tokens, and are input and output as separate tracks that are fused into a single latent space within the model. ESM3 is trained with a generative masked language modeling objective:
ME: Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - ESM3 reasons over three main aspects: the sequence, structure, and function of proteins.
$$ \mathcal{L}=-\mathbb{E}{x, m}\left[\frac{1}{|m|} \sum{i \in m} \log p\left(x{i} \mid x{\backslash m}\right)\right] $$
A random mask $m$ is applied to the tokens $x$ describing the protein, and the model is supervised to predict the identity of the tokens that have been masked. During training, the mask is sampled from a noise schedule so that ESM3 sees many different combinations of masked sequence, structure, and function, and predicts completions of any combination of the modalities from any other. This differs from the classical masked language modeling (28) in that the supervision is applied across all possible masking rates rather than a single fixed masking rate. This supervision factorizes the probability distribution over all possible predictions of the next token given any combination of previous tokens, ensuring that tokens can be generated in any order from any starting point (29-31).
ME: Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - A random mask $m$ is used on the protein tokens $x$.
To generate from ESM3, tokens are iteratively sampled. Starting from a sequence of all mask tokens, tokens can be sampled one at a time, or in parallel, in any order, until all tokens are fully unmasked (Fig. 1A). Masking is applied independently to sequence, structure, and function tracks, which enables generation from any combination of empty, partial, or complete inputs. ESM3's training objective is also effective for representation learning. We choose a noise schedule that balances generative capabilities with representation learning (Appendix A.2.2). Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - ESM3 uses an iterative sampling method for token generation.
Tokenization enables efficient reasoning over structure. Protein structures are tokenized by a discrete auto-encoder (32), which is trained to compress the high dimensional space of three-dimensional structure into discrete tokens (Fig. 1C). We propose an invariant geometric attention mechanism to efficiently process three-dimensional structure. The mechanism operates in local reference frames defined by the bond geometry at each amino acid, and allows local frames to interact globally through a transformation into the global frame (Appendix A.1.6). This mechanism can be efficiently realized through the same computational primitives as attention (33), and is readily scalable. The local structural neighborhoods around each amino acid are encoded into a sequence of discrete tokens, one for each amino acid.
ME: Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - Tokenization helps in efficient reasoning over structure.
When predicting or generating protein structure, structure tokens output by ESM3 are passed to the decoder, which reconstructs the all-atom structure. The autoencoder is trained to encode and reconstruct atomic coordinates with a geometric loss that supervises the pairwise distances and relative orientations of bond vectors and normals (Appendix A.1.7.3.1). This tokenization delivers nearperfect reconstruction of protein structure ( $<0.3 \AA$ RMSD on CAMEO, Fig. S3), enabling representation of structure at the input and output with atomic accuracy. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - Structure tokens output by ESM3 are used in the prediction or generation of protein structures.
We also find that providing ESM3 direct access to atomic coordinates in the input via a geometric attention projection into the transformer improves the response to atomic coordinate prompts. ESM3 can be conditioned on either or both of tokenized structure and atomic coordinates. We supplement these structure representations with coarse grained tokens encoding secondary structure state (SS8) and solvent accessible surface area (SASA). Function is presented to the model in the form of tokenized keyword sets for each position in the sequence. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - ESM3's ability to respond to atomic coordinate prompts is enhanced by providing it with direct access to atomic coordinates through a geometric attention projection.
ESM3 is a bidirectional transformer. While extensive research has gone into creating specialized architectures and training objectives for proteins, we find that tokenization paired with a standard masked language modeling objective and the basic transformer architecture is highly effective for both representation learning and generative modeling. Sequence, structure, and function tracks are input as tokens, which are embedded and fused, then processed through a Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - ESM3 is a bidirectional transformer.
Please explain to a person without a background in LLMs
AI: Certainly! Let's delve into this complex topic in a way that doesn't require prior knowledge of Large Language Models (LLMs) or specialized biological knowledge.
What is ESM3?
ESM3 is essentially a sophisticated computer program designed to understand and predict various aspects of proteins - which are crucial molecules for life. It can look at the sequence of a protein (the order of its building blocks), its three-dimensional structure, and its function (what it does in the body) in a unique and interconnected way. Think of ESM3 as a very smart biologist that has been trained to analyze proteins in great depth.
How does it work?
The way ESM3 learns is similar to someone trying to complete a puzzle with some of the pieces hidden - it tries to guess the missing pieces based on the overall picture. In technical terms, it gets trained by being shown proteins with some parts 'masked' or hidden, and it learns by trying to predict these hidden parts. This is done over and over with different parts hidden each time, which helps it understand how proteins are built and function at a deep level.
Why is it special?
Understanding Proteins Holistically: ESM3 considers the sequence, structure, and function of proteins all at once. Imagine you only knew what different car parts looked like but had no idea how these parts fit together or what they do. ESM3, on the other hand, understands not only what the parts are but also how they come together and what they're supposed to do - a much fuller understanding of the 'mechanics' of life at the molecular level.
Flexibility in Learning and Prediction: It's trained to predict any part of the protein from any other part, giving it a flexible understanding. This means it can work with incomplete information to make predictions, much like filling in missing words in a sentence based on the context provided by the rest of the sentence.
Protein Structure: Understanding the 3D shape of proteins is crucial because a protein's shape determines its role and function. ESM3 is trained to 'visualize' and predict this structure by learning from thousands of known protein shapes. It does this by breaking down the complex shape into simpler, easier-to-understand pieces.
High Accuracy: This artificial intelligence can predict the structure and function of proteins with remarkable precision, aiding scientists in understanding how different proteins work and interact - potentially accelerating drug discovery and the understanding of diseases.
In essence, ESM3 is like a highly advanced tool for scientists, helping them decipher the complex language of proteins much faster and more accurately than traditional methods. It's a blend of biology, computer science, and mathematics, all working together to unlock the secrets of life at the molecular level.
ME: Please explain for an expert in LLMs
AI: For an expert in Large Language Models (LLMs), delving into the intricacies of ESM3 offers a fascinating insight into the application of LLM methodologies to the domain of protein sequences, structures, and functions. ESM3 represents an avant-garde approach in blending traditional natural language processing (NLP) techniques with biological data to address complex bioinformatics challenges.
Model Architecture and Training Regime
At its core, ESM3 leverages a bidirectional transformer architecture, akin to those used in leading LLMs like BERT or GPT, but with critical modifications tailored to handle the multimodal nature of protein data. The model ingests sequences as tokens, analogous to words or characters in NLP, but these tokens represent amino acids, structure descriptors, or functional attributes rather than textual information.
The training objective is a generative masked language modeling (MLM) task, yet it innovates by incorporating a dynamic masking strategy. Instead of a static mask rate (e.g., masking 15% of tokens as in BERT), ESM3 employs a varied masking approach where the mask ratio changes, allowing the model to be challenged with a wider range of prediction tasks during training. This dynamic masking encourages a more robust internal representation of protein-related data, enhancing the model's ability to interpolate or extrapolate from given sequences, structural data, or functional annotations.
Tokenization of Structure and Function
A unique aspect of ESM3 is how it deals with protein structure. Utilizing a discrete autoencoder, the model compresses 3D structural information into discrete tokens. This process necessitates an efficient method to capture the complexity of protein structures in a format that's amenable to transformer-based models. The introduction of an invariant geometric attention mechanism, which enables the model to interpret 3D structures in their local spatial contexts and integrate these insights globally, is particularly innovative. This mechanism mirrors attention mechanisms in LLMs but adapts them to handle the inherently geometric nature of protein data.
For functional data, ESM3 incorporates tokenized keyword sets, allowing the model to understand and predict the functional aspects of proteins based on amino acid sequences and structural configurations. This multimodal tokenization strategy facilitates a comprehensive understanding across sequence, structure, and function.
Generative Capabilities and Representation Learning
ESM3's generative capabilities are highlighted by its ability to predict missing information across any of the input modalities, whether starting from scratch or from partial inputs. This aspect is particularly important for tasks such as de novo protein design or function prediction, where incomplete information must often be reasoned into coherent wholes.
The balance it strikes between generative capabilities and representation learning is notable. By selecting an appropriate noise schedule, ESM3 improves both its predictive power and the quality of the latent representations it learns. This dual capacity makes it not just a tool for generating predictions but also a powerful method for exploratory data analysis and hypothesis generation in the protein space.
Conclusion
For LLM experts, ESM3 represents an exciting exploration into how the principles underlying successful language models can be extended beyond text to tackle problems in biological sequence, structure, and function prediction. Its training methodology, attention to the uniqueness of protein structures, and innovative use of tokenization position ESM3 as a state-of-the-art model that pushes the boundaries of what's possible with LLM technologies in the life sciences.
Figure 1. ESM3 is a generative language model that reasons over the sequence, structure, and function of proteins. (A) Iterative sampling with ESM3. Sequence, structure, and function can all be used to prompt the model. At each timestep $\mathrm{t}$, a fraction of the masked positions are sampled until all positions are unmasked. (B) ESM3 architecture. Sequence, structure, and function are represented as tracks of discrete tokens at the input and output. The model is a series of transformer blocks, where all tracks are fused within a single latent space; geometric attention in the first block allows conditioning on atomic coordinates. ESM3 is supervised to predict masked tokens. (C) Structure tokenization. Local atomic structure around each amino acid is encoded into tokens. (D) Models are trained at three scales: 1.4B, 7B, and 98B parameters. Negative log likelihood on test set as a function of training FLOPs shows response to conditioning on each of the input tracks, improving with increasing FLOPs. (E) Unconditional generations from ESM3 98B (colored by sequence identity to the nearest sequence in the training set), embedded by ESM3, and projected by UMAP alongside randomly sampled sequences from UniProt (in gray). Generations are diverse, high quality, and cover the distribution of natural sequences. stack of transformer blocks. The first transformer block also includes a geometric attention layer for atomic structure coordinate conditioning. At the output of the model, shallow MLP heads project the final layer representation into token probabilities for each of the tracks. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - ESM3 is a generative language model designed for predicting protein sequences, structures, and functions.
The largest ESM3 model is trained on 2.78 billion natural proteins derived from sequence and structure databases (2, 34-37). As a small fraction of structures have been experimentally determined relative to sequences, we leverage predicted structures $(4,5)$. We also generate synthetic sequences with an inverse folding model (described in Appendix A.2.1.3) for all structures, including predicted ones. Function keywords are derived by predicting functional annotations from sequence using a library of hidden markov models (38). Overall this increased training data to 3.15 billion protein sequences, 236 million protein structures, and 539 million proteins with function annotations, totaling 771 billion unique tokens. Full details of the training dataset are described in Appendix A.2.1.8. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - The largest ESM3 model is trained on 2.78 billion natural proteins.
Please analyze this paragraph from the point of view of an expert scientist and be critical of any omissions or possible sources of problems you see.
AI: Analyzing this paragraph from the perspective of an expert scientist specializing in computational biology and protein modeling reveals several points of interest and potential areas for critique:
The text notes that the model leverages predicted structures due to the relatively small number of experimentally determined structures. While advances in protein structure prediction, notably those achieved by tools like AlphaFold, have significantly reduced the gap between predicted and experimental structures, there are inherent inaccuracies and biases in prediction models that can propagate through the ESM3 model. This reliance on predicted structures could introduce systemic errors or biases, particularly for proteins with novel folds or those from less-studied organisms.
The generation of synthetic sequences through an inverse folding model is an innovative way to enhance the training data. However, the quality and diversity of these synthetic sequences are crucial. There is a risk that the synthetic sequences may not accurately reflect natural protein diversity, especially for proteins from extremophiles or those undergoing rapid evolution. The method's robustness in generating realistic and biologically plausible sequences should be scrutinized.
Protein function prediction using a library of hidden Markov models is a well-established method. However, functional annotations in databases often suffer from incompleteness, errors, or biases toward well-studied proteins. The approach to predicting function keywords from sequences may also inherit these biases, affecting the model's ability to accurately predict functions for less-characterized or novel proteins.
While the large scale of the dataset (3.15 billion protein sequences, 236 million structures, and 539 million function annotations) is impressive, the quality of this data is as critical as its quantity. The paragraph does not detail the steps taken to ensure data quality, such as filtering out redundant or low-quality sequences and structures, which could significantly impact the model's performance.
The diversity of the training dataset directly influences the model's generalizability. The paragraph does not mention the taxonomic or environmental diversity of the source data. Models trained predominantly on data from model organisms may not perform as well on sequences from less-studied species.
The paragraph focuses on the dataset's preparation and lacks discussion on how the model's performance was evaluated, particularly concerning the novel synthetic sequences and predicted functions. Insights into the validation methods, such as cross-validation with experimentally determined structures and functions, would be vital to assess the reliability of the model's predictions.
Given the model's extensive training data, there is a potential risk of overfitting, where the model may excel at predicting features of the training data but perform poorly on unseen data. Strategies to mitigate overfitting, such as regularization techniques or training with a diverse set of sequences, should be examined.
Finally, while not a technical critique, it's important to consider the ethical implications of creating and using powerful predictive models like ESM3. Issues around data privacy, consent (particularly for human-derived proteins), and potential misuse for creating harmful biological agents should be addressed.
In summary, while the development of the ESM3 model represents a significant advance in protein structure and function prediction, the methodologies employed warrant close scrutiny to ensure the reliability, accuracy, and ethical use of the generated predictions and synthetic sequences.
We train ESM3 models at three scales: 1.4 billion, 7 billion, and 98 billion parameters. In an initial series of experiments to evaluate representation learning performance in response to architecture hyperparameters, we find a greater response to increasing depth than to width. This informed the choice of relatively deep networks for the final architectures, with the 98 billion parameter model incorporating 216 Transformer blocks (Appendix A.1.5). Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - The ESM3 models were trained at three different scales: 1.4 billion, 7 billion, and 98 billion parameters.
Scaling ESM3 from 1.4 billion to 98 billion parameters results in substantial improvements in the validation loss for all tracks, with the greatest improvements observed in sequence loss (Fig. 1D, Fig. S11). These gains in validation loss lead to better representation learning (Table S7 and Fig. S8). In single sequence structure prediction (Table S8) on CAMEO, ESM3 98B obtains 0.895 mean local distance difference test (LDDT) and surpasses ESMFold (0.865 LDDT). Unconditional generation produces high-quality proteins-with a mean predicted LDDT (pLDDT) 0.84 and predicted template modeling score (pTM) 0.52-that are diverse in both sequence (mean pairwise sequence identity 0.155 ) and structure (mean pairwise TM score 0.48 ), spanning the distribution of known proteins (Fig. 1E, Fig. S13). Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - Scaling ESM3 from 1.4 billion to 98 billion parameters results in substantial improvements in the validation loss for all tracks.
Please analyze this paragraph from the point of view of an expert scientist and be critical of any omissions or possible sources of problems you see.
AI: Analyzing the paragraph provided from the viewpoint of an expert scientist, a number of critical reflections and potential issues can be brought up regarding omissions and sources of problems related to the dramatic scaling of parameters in the ESM3 model from 1.4 billion to 98 billion, as well as the results discussed therein.
Computational Resources and Accessibility: The increase in parameters from 1.4 billion to 98 billion undoubtedly requires significant computational resources. This might limit the accessibility of such a model to a few well-funded laboratories or institutions, curbing the democratization of advanced computational tools in the scientific community. There's an omission of any discussion on the computational costs and energy consumption, which are essential factors to consider given the increasing awareness of the environmental impact of large-scale computations.
Overfitting Risk: While the paragraph notes substantial improvements in validation loss, there is no mention of how the model performs on completely unseen data, which is a standard measure to assess if the model generalizes well or if it is overfitting. The mention of "validation loss" improvements does not necessarily imply that the model will perform equally well in real-world applications or on datasets that were not part of the training/validation sets.
Underrepresented Data: The scaling of parameters typically enables a model to learn finer details and nuances in the data. However, it can also exacerbate biases towards overrepresented features in the training set. The paragraph omits any discussion on the diversity and representativeness of the training dataset, which are crucial for assessing the reliability and applicability of the model's predictions across diverse biological entities.
Statistical Significance and Error Margins: The increases in performance metrics such as LDDT and TM scores are mentioned without any accompanying statistical analysis data. For a scientifically rigorous evaluation, it would be essential to know the confidence intervals or p-values associated with these improvements to understand if the differences are statistically significant and not due to chance variations in the dataset.
Comparison to Baselines and State of the Art: While the paragraph mentions that ESM3 98B surpasses ESMFold in single sequence structure prediction, it lacks a broader comparison to other state-of-the-art models in the field. A critical understanding of how ESM3 98B positions itself within the landscape of current bioinformatic tools and whether it addresses limitations of existing models is missing.
Diversity and Quality of Generated Proteins: The paragraph asserts that the model produces high-quality, diverse proteins but does not specify how the diversity and quality were assessed or benchmarked. The mean pairwise sequence identity and TM score are given, yet there is no contextualization of these figures within existing literature or experimental validation of the produced structures.
The paragraph omits any discussion on the ethical and practical implications of deploying a substantially more powerful model like ESM3 98B. For instance, issues related to biosecurity, the potential creation of harmful proteins, or the monopolization of cutting-edge tools by a few entities are not addressed.
In summary, while the scaling of the ESM3 model to 98 billion parameters appears to offer significant improvements in certain metrics, a critical analysis suggests several areas where additional information, broader context, and a more rigorous scrutiny of the results and their implications would be necessary to fully evaluate the advancements and their potential impact on the field of bioinformatics and beyond.
We explore the ability of ESM3 to follow complex prompts with different compositions. ESM3 can be prompted with instructions from each of its input tracks: sequence, structure coordinates, secondary structure (SS8), solvent-accessible surface area (SASA), and function keywords. This allows prompts to be specified at multiple levels of abstraction, from atomic level structure to high level keywords describ- ing the function and fold topology, using the learned generative model to find a coherent solution that respects the prompt. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - ESM3 can be prompted with instructions from various input tracks.
Please analyze this paragraph from the point of view of an expert scientist and be critical of any omissions or possible sources of problems you see.
AI: The paragraph provides an overview of the capabilities of ESM3, ostensibly a computational model designed for protein prediction tasks, leveraging multi-input tracks such as sequence, structure coordinates, secondary structure (SS8), solvent-accessible surface area (SASA), and function keywords. The claim is that ESM3 can follow complex prompts across these various dimensions to generate coherent predictions. While promising, there are several critical aspects and potential problems that need to be highlighted from an expert standpoint:
Model Generalizability: One of the core challenges in protein prediction models involves their generalizability to unseen or less common protein structures. The paragraph does not address how ESM3 handles this issue, particularly regarding rare folds or proteins from less-studied organisms. The ability of a model to generate coherent solutions across a diverse set of protein structures is vital for its application in real-world scenarios.
Data Dependency: The performance of machine learning models, such as ESM3, is highly dependent on the quality and diversity of the training data. There's no mention of the data used to train ESM3 or how issues of bias or incomplete datasets were addressed. Incomplete or biased datasets could lead to incorrect predictions, particularly for proteins that do not fall within the "typical" range of the training data.
Interpretability and Validation: While the paragraph suggests that ESM3 can integrate prompts across various levels of abstraction, it does not mention how the model's predictions can be interpreted or validated. For example, how does one assess the accuracy of the generated structure or function predictions? The ability to critically evaluate and trust the model's output is crucial for scientific and practical applications.
Computational Requirements: Models that integrate multiple input tracks and produce complex outputs tend to be computationally intensive. The paragraph does not address the computational resources required for running ESM3, which is a critical factor for its accessibility and use in different environments, ranging from academic labs to industry settings.
Handling of Dynamic and Flexible Regions: Proteins are dynamic entities, and their function often depends on conformational flexibility. The text does not discuss how ESM3 deals with intrinsically disordered regions or proteins that adopt multiple stable conformations, which are challenging for many prediction models.
Integration with Experimental Data: The ultimate test of any protein prediction model is its ability to generate hypotheses or predictions that can be validated experimentally. The paragraph does not mention if or how ESM3's predictions can be integrated with experimental workflows, a critical aspect for its adoption in drug discovery, functional genomics, and other areas.
Ethical Considerations and Misuse: Finally, while not directly related to the scientific capabilities of ESM3, any powerful predictive tool raises questions regarding ethical use, potential misuse, and the consequences of incorrect predictions. These aspects are increasingly important to address in the development and deployment phases of computational tools.
In summary, while ESM3 appears to offer a versatile and potentially powerful tool for protein prediction, several areas require clarification and further discussion, including its generalizability, data requirements, interpretability, computational demands, handling of protein dynamics, integration with experimental validation, and broader ethical considerations.
We evaluate ESM3's ability to follow prompts in each of the tracks independently. A set of prompts are constructed for each of the tracks using a temporally held out test set of natural proteins (Appendix A.3.7). We evaluated the resulting generations for consistency with the prompt and foldability, the confidence of the structure prediction TM-score (pTM) under ESMFold. We define consistency metrics for each track: constrained site RMSD (cRMSD) is the RMSD between the prompt coordinates and the corresponding coordinates in the generation; SS3 accuracy is the fraction of residues where three-class secondary structure between the prompt and generations match; SASA spearman $\rho$ is the correlation between the SASA prompt and the corresponding region of the generation; keyword recovery is the fraction of prompt keywords recovered by InterProScan (38). Across all tracks, ESM3 finds solutions that follow the prompt, and have confidently predicted structures by ESMFold (pTM $>0.8$ ) (Fig. 2A). Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - Evaluation of ESM3's ability to follow prompts is conducted for multiple tracks independently.
AI: This summary describes a systematic evaluation of ESM3, a presumably advanced model trained to generate protein structures that are consistent with specific constraints or prompts given to it. The evaluation process aims to assess how well ESM3 can generate protein structures that not only follow the instructions provided in various prompts but also result in viable, foldable proteins as predicted by ESMFold, a tool for predicting the confidence of protein structure predictions.
The methodology for this evaluation involved crafting a series of prompts divided into different tracks, each focusing on distinct aspects or requirements of protein structure modeling. These prompts were based on a dataset of natural proteins, specifically chosen from data that was not used during the model's training phase to ensure an unbiased assessment of its generative abilities.
Key metrics were defined to quantitatively measure the model's performance in adhering to the prompts and generating structurally viable proteins, namely:
The findings indicate that across all tracks, ESM3 was able to generate protein structures that not only adhered to the detailed instructions provided in the prompts but also were deemed foldable with high confidence (pTM score greater than 0.8) by ESMFold. Figure 2A in the context presumably illustrates these results, offering visual or quantitative evidence of ESM3's proficiency in creating accurate and reliable protein structures based on specified criteria. This highlights ESM3's potential as a powerful tool in protein design and modeling, allowing for the custom generation of proteins with desired structures and functions.
ME: please explain it for an expert in LLMs and AI
AI: The evaluation of ESM3's capabilities in generating natural protein structures from specified prompts showcases a significant advancement in the field of artificial intelligence (AI), specifically within the domain of language models and their application in structural biology. ESM3 appears to be a state-of-the-art iteration of generative AI models, tailored for the complex task of protein generation, exhibiting an interdisciplinary leap where natural language processing methodologies are adapted to understand and predict the folding and structural intricacies of proteins.
The evaluation methodology detailed leverages a diverse set of metrics, each meticulously designed to assess the fidelity of ESM3's generated outputs against a series of highly detailed and nuanced prompts. These metrics are:
Constrained Site RMSD (cRMSD): Serving as a precise spatial accuracy metric, cRMSD evaluates how closely the model-generated structures match the specified atomic coordinates in the prompts. This is crucial in applications where specific atomic arrangements are essential for the biological function or stability of the protein.
SS3 Accuracy: This metric aligns with traditional classification tasks in AI but is applied here to the prediction of secondary structures (helices, sheets, coils), assessing the model's ability to understand and recreate the local structural motifs of proteins based on textual or encoded prompts.
SASA Spearman ρ: Reflecting a regression-based correlation measurement, this evaluates the model's proficiency in capturing and replicating the solvent accessibility of amino acids as specified in the prompts. Solvent accessibility is a key factor influencing protein stability and interaction capabilities, making this metric indicative of the model's understanding of protein dynamics and functional properties.
Keyword Recovery: This metric examines the model's ability to interpret and incorporate functional or structural keywords from the prompts into the generated protein structures. Utilizing InterProScan for validation, this metric bridges the gap between textual description and functional/structural protein annotation, akin to evaluating a language model's understanding of context and its ability to generate relevant content based on that context.
The findings, particularly the achievement of confidently predicted structures (pTM > 0.8), not only underscore ESM3's proficiency in generating biologically plausible protein structures but also highlight the model's understanding of complex structural biology concepts distilled into prompt-based instructions. This implies a significant abstraction capability of the model, where it not only interprets but effectively applies sophisticated biological principles to generate functional protein structures.
Moreover, the successful application of ESM3 in this domain represents a promising amalgamation of AI, specifically LLMs, into computational biology and structural bioinformatics. It reflects the growing trend of utilizing AI's pattern recognition and generative capabilities to tackle challenges in scientific domains that are traditionally considered beyond the scope of existing AI methodologies. This not only expands the utility of LLMs into new areas but also opens avenues for innovative approaches to protein design, with potential implications for drug discovery, enzyme design, and understanding disease mechanisms at the molecular level.
Unconditional generations reflect the distribution of natural proteins. Since we observed ESM3 can faithfully follow prompts, we reasoned that prompting could steer the model to generate proteins that differ from natural proteins. First we test the ability of the model to follow out-of-distribution prompts. We construct a set of prompts combining SS8 and SASA from held out structures (TM $<0.7$ to training set). Under these prompts, while the model continues to generate coherent globular structures (mean pTM $0.85 \pm 0.03$ ), the distribution of similarities to the training set (as measured by TM-score and sequence identity) shifts to be more novel (average sequence identity to nearest training set protein $<20 \%$ and mean TM-score $0.48 \pm 0.09$; Fig. 2B top). To test the ability to generalize to structures beyond the distribution of natural proteins, we use secondary structure prompts derived from a dataset of artificial symmetric protein designs distinct from the natural proteins found in the training dataset (Appendix A.3.8). Similarly, ESM3 produces high confidence generations (pTM $>0.8$, pLDDT $>0.8$ ) with low sequence and structure similarity to proteins in the training set (sequence identity $<20 \%$ and TM-score $0.52 \pm 0.10$; Fig. 2B bottom), indicating that the model can be used to generate protein sequences and structures highly distinct from those that exist in nature. Please analyze this paragraph from the point of view of an expert scientist and be critical of any omissions or possible sources of problems you see. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: Analyzing the paragraph from the perspective of an expert scientist, while the paragraph provides valuable insights into the capabilities of the model (ESM3) in generating novel protein structures, there are inherent criticisms and potential problems not addressed. Below, the key facts and ideas are extracted and outlined in markdown list format. Subsequently, I will highlight potential omissions and sources of problems.
Addressing these omissions and potential problem areas would provide a more comprehensive understanding of the model's efficacy, limitations, and potential impact on protein engineering and synthetic biology endeavors.
AI: The paragraph presents intriguing advancements in the ability of a deep learning model, referred to as ESM3, to generate protein structures. The focus here is on ESM3's ability to respond to prompts and generate proteins both within and beyond the distribution of natural proteins. I'll dissect the paragraph from a critical scientific standpoint, underlining significant insights while also highlighting potential areas of concern or omission.
Out-of-Distribution Generalization: The paragraph underscores ESM3's promising performance in generating coherent globular structures from out-of-distribution prompts, which is a significant milestone in protein design. The ability to steer away from the distribution of known proteins and still maintain structural integrity in the generated proteins is notable.
High Confidence in Novel Generations: Both sets of experiments show that ESM3 can create structures with high confidence (high pTM and pLDDT scores), even when these generations have low similarity to known proteins. This is crucial for novel protein design, potentially paving the way for breakthroughs in creating proteins with unprecedented functionalities.
Evaluation of Functional Viability: While the focus on structural metrics like pTM, pLDDT, and sequence identity is important, the paragraph does not address the functional viability or stability of these generated proteins. Structural coherence does not guarantee functional efficacy, and additional analyses (e.g., molecular dynamics simulations, experimental validation) are necessary to ascertain that these novel proteins can be viably expressed and are functionally active.
Generalization beyond Globular Proteins: The reported experiments centered on globular proteins. However, proteins encompass a wider range of structures, including fibrous and membrane proteins. The model's ability to generalize across different types of proteins, which may have distinct folding patterns and functional roles, remains unaddressed.
Prompt Dependency and Limitations: The paragraph illustrates ESM3's ability to follow prompts effectively but does not discuss the limitations of this approach. For instance, the design of prompts requires substantial understanding of protein structures and might be limiting if the prompts do not accurately reflect feasible structural domains or functional sites.
Dataset Bias and Diversity: Artificial symmetric protein designs are distinct from natural protein databases, facilitating novel protein generation. However, the diversity and representativeness of these artificial designs need scrutiny. Biases in the artificial designs or their insufficient diversity could limit the generalization capabilities of ESM3, skewing toward certain structural motifs or topologies.
Scalability and Computational Costs: Implementing deep learning models for protein generation, especially for out-of-distribution proteins, likely incurs high computational costs. The scalability of ESM3's approach to larger datasets or more complex protein designs is not mentioned, which could be a significant constraint in practical applications.
Ethical and Biosafety Considerations: Lastly, generating proteins beyond the natural distribution raises ethical and biosafety concerns that are not addressed. The potential for creating harmful proteins, either unintentionally or through misuse, necessitates a broader discussion on the control measures and ethical guidelines governing the use of such technologies.
The paragraph presents compelling progress in the field of protein design using deep learning, showcasing ESM3's ability to generate novel protein structures effectively. Despite these advances, there remain critical areas for improvement and consideration, spanning functional validation, model generalization capabilities, application range, and ethical implications. Addressing these aspects would be crucial for translating technological advances into safe and practical applications.
ESM3 is able to follow complex prompts, and has the ability to compose prompts from different tracks, and at different levels of abstraction. To evaluate this ability, we prompt ESM3 with motifs that require the model to solve for spatial coordination of individual atoms, including ones requiring tertiary coordination between residues far apart in the sequence, such as catalytic centers and ligand binding sites. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - ESM3 can follow complex prompts.
Please explain how this is breakthrough
AI: The aspect of ESM3 (which stands in this context for a hypothetical or future version of an Evolutionary Scale Modeling system, third iteration) that allows it to follow complex prompts and solve for spatial coordination of individual atoms, including tertiary coordination between residues far apart in the sequence, represents a significant breakthrough in computational biology for several reasons:
Understanding Complex Protein Structures: Protein functions are determined by their three-dimensional structures. The ability to accurately model and predict tertiary and quaternary structures, including the precise spatial coordination of atoms and the folding patterns of amino acid chains, is crucial for understanding how proteins work. This breakthrough implies a level of detail and accuracy that was previously difficult or impossible to achieve, especially for proteins with complex folding patterns and those that require precise coordination of distant residues for their function.
Accelerating Drug Discovery: Knowing the exact arrangement of atoms within catalytic centers and ligand binding sites can dramatically accelerate drug discovery and design. By accurately predicting how proteins fold and where key binding sites are located, researchers can more efficiently design or identify compounds that can specifically interact with these sites to modulate protein function. This can lead to the development of new drugs with higher efficacy and fewer side effects.
Multiscale Modeling: The ability to compose prompts at different levels of abstraction indicates a versatile tool capable of integrating information across different biological scales, from the atomistic to the macromolecular. This multiscale modeling is crucial for understanding how changes at the atomic level (such as mutations that alter a single amino acid) can affect the overall structure and function of a protein, and by extension, impact cellular processes and organismal health.
Overcoming Sequence Distance Constraint: Traditional methods often struggle with residues that are far apart in the linear sequence but come close together in the folded structure to form functional sites. The ability to model tertiary coordination between such residues is a significant step forward, enabling the prediction of protein structures and interactions that were previously challenging due to the limitations in understanding long-range interactions.
Applications Beyond Protein Folding: While the immediate application is in protein structure prediction and analysis, the capabilities of ESM3 extend to other areas such as enzyme design, understanding disease mechanisms at the molecular level (e.g., how mutations disrupt normal protein function), and synthetic biology. This opens up new possibilities for the biotechnology and pharmaceutical industries, among others.
Technological Innovation: To achieve this level of detail and complexity in predictions, significant innovations in machine learning, data handling, and computational power are required. This denotes not just a breakthrough in understanding biological systems but also in the technological and algorithmic approaches used to model such systems.
In summary, ESM3's ability to accurately model the spatial coordination of atoms within proteins, particularly for complexes and distant residues, signifies a leap forward in computational biology and protein science. It enhances our ability to understand life at a molecular level and holds promise for numerous practical applications in medicine, biotechnology, and related fields.
A
Brompting for out-of-distribution folds
Figure 2. Generative programming with ESM3. (A) ESM3 can follow prompts from each of its input tracks. Density of faithfulness to prompting for each of the tracks is shown. Generations achieve consistency with the prompt and high foldability (pTM). (B) ESM3 can be prompted to generate proteins that differ in structure (left) and sequence (right) from natural proteins. Prompted generations (blue) shift toward a more novel space vs. unconditional generations (red), in response to prompts derived from out-of-distribution natural structures (upper panel) and computationally designed symmetric proteins (lower panel). (C) ESM3 generates creative solutions to a variety of combinations of complex prompts. We show compositions of atomic level motifs with high level instructions specified through keyword or secondary structure. Fidelity to the prompt is shown via similarity to reference structure (for keyword prompts) and all-atom RMSD to the prompted structure (for atomic coordination prompts). Solutions differ from the scaffolds where the motif was derived (median TM-score $0.36 \pm 0.14$ ), and for many motifs (e.g. serotonin, calcium, protease inhibitor, and Mcl-1 inhibitor binding sites), we could find no significant similarity to other proteins that contain the same motif. (D) An example of especially creative behavior. ESM3 compresses a serine protease by $33 \%$ while maintaining the active site structure. Please analyze this paragraph from the point of view of an expert scientist and be critical of any omissions or possible sources of problems you see. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: Analyzing the paragraph critically reveals several interesting facts and ideas about the capabilities of ESM3, but also raises questions regarding omissions or areas where challenges might occur. Below is a deeper dive into each fact or idea extracted from the paragraph:
In summary, while the abilities of ESM3 as described offer promising directions for protein engineering and synthetic biology, the analysis could benefit from a more nuanced discussion on functional validation, real-world applicability, computation limitations, and broader implications of creating novel proteins
AI: Analyzing the provided text critically from an expert scientist's perspective, several points merit attention regarding omissions and potential sources of problems. The paragraph effectively communicates impressive capabilities of ESM3 in generative programming for protein design, yet some aspects could benefit from further specificity or clarification.
Input Tracks and Prompting Details: While it mentions that ESM3 can follow prompts from its input tracks, the paragraph does not specify what these input tracks entail or how they are designed. The methodology behind the construction of these prompts and the specification of "density of faithfulness" could be more transparent. This information is crucial for reproducibility and understanding how ESM3 interprets and prioritizes various prompts.
Benchmarking and Measurements: The paragraph references measurements like high foldability (pTM), similarity to reference structure, and all-atom RMSD without offering a clear definition or the rationale behind choosing these specific metrics. Furthermore, the context of what values are considered high or low for these metrics in the field of protein design is missing.
Benchmark Against Current Standards: There is a lack of direct comparison to existing generative models or protein design algorithms. While the novelty is implied through the generation of proteins differing in structure and sequence from natural proteins, a more nuanced discussion on how ESM3 improves or differs from current methods could strengthen its stated contributions.
Representation of Novel Space: The paragraph indicates ESM3's ability to generate proteins that shift toward a "more novel space" but does not elaborate on how this novelty is quantified or its implications for practical applications, such as drug design or biotechnology. A more in-depth examination of the practical utility of these novel proteins would be beneficial.
Fidelity and Functionality: The paragraph lauds ESM3 for generating creative solutions with high fidelity to prompts; however, it omits any discussion on the biological functionality or viability of these generated proteins. The structural integrity (e.g., maintaining the active site structure of a serine protease) is noted, but the actual biological efficacy or stability of these proteins in a real-world setting is not addressed.
Comparison with Motif Databases: While the text claims that for many motifs, no significant similarity to other proteins containing the same motif could be found, it lacks a detailed explanation of how this analysis was conducted. Details on the databases or datasets used for comparison, the thresholds for significance, and the implications of these findings on the field of protein engineering would be pertinent.
The paragraph presents a compelling view of ESM3's potential in generative protein design but could benefit from a more rigorous presentation of methods, clearer benchmarks against existing solutions, detailed explanations of novel findings, and an in-depth discussion of potential applications and limitations. Acknowledging these areas could pave the way for further research, enhancing the scientific value and applicability of ESM3 in protein engineering and beyond.
ME: please explain it in detail for an expert
AI: Analyzing the given text from the perspective of an advanced professional in the field of computational biology or protein engineering, several refinements and clarifications are warranted to elevate the completeness and scientific rigor of the report:
Input Track Specification: The document lacks detail on the structure and formation of input tracks used to prompt ESM3. In generative models, especially those using deep learning, the design of input prompts critically influences the model's output. The "density of faithfulne ssto prompting" is an intriguing concept, suggesting a quantifiable metric for prompt adherence, but without a clear methodology for measuring density or an explanation of input track composition, evaluating the model's fidelity to these prompts becomes challenging.
Metrics of Evaluation: Metrics like pTM (presumably predicting the Threshold Model) for foldability and RMSD (Root-Mean-Square Deviation) for structural similarity are well-established in structural biology. However, their application without a detailed exposition of baseline thresholds, calculation methods, or comparison standards leaves their implications ambiguous. For example, the document should specify the pTM scores that correlate with high foldability or what RMSD values were considered significant for fidelity assessment.
Comparative Performance Analysis: The absence of a comparative analysis with existing protein design models (such as Rosetta or AlphaFold) on similar tasks limits the contextualization of ESM3's performance. For novel protein generations, especially, an explicit comparison in terms of structural novelty, function, and design efficiency could significantly bolster the claim of superiority or novelty of ESM3's approach.
Definition and Quantification of Novelty: The paragraph notes ESM3's ability to generate proteins that explore a "more novel space" without clearly defining what constitutes novelty in this context or how it's quantified. In protein engineering, novelty could refer to unprecedented sequences, structures, functions, or a combination of these factors. A direct discussion on how ESM3 specifically fosters novelty -- whether through divergent sequence generation, folding patterns, or functional motifs -- would provide deeper insights into the model's capabilities and limitations.
Functional Viability of Generated Proteins: While ESM3's technical prowess in generating structurally consistent and novel proteins is highlighted, there is scant information on the biological functionality or stability of these proteins. Successful protein design is not only a matter of generating structurally innovative proteins but also ensuring they are biologically active and stable. Discussions surrounding the expression viability, functional assays, or in vitro/in vivo tests of generated proteins are crucial to validate the utility of these designs in practical scenarios.
Database Comparisons for Motif Originality: Claims of generating motifs with no significant similarity to known proteins necessitate a detailed explanation of the databases, algorithms, and similarity thresholds used for this analysis. The omission of such details raises questions regarding the breadth of the comparison (e.g., considering only a specific database or a wide array of protein databases), the criteria for motif similarity, and how "significance" is quantified in this context.
While the document succinctly outlines the generative capabilities of ESM3 in protein design, a more rigorous and detailed presentation would enhance its scientific value. Addressing the specifics of methodology, benchmarking, functional assessment, and explicit examples with experimental validation are fundamental to comprehensively evaluate ESM3's potential impact on computational biology and protein engineering fields. Such a critique aims to encourage not just the refinement of reporting but also the advancement of the methodologies underpinning this exciting field of research.
We combine these with prompts that specify the fold architecture. For each unique combination of motif and scaffold, we generate samples until the prompt is satisfied (cRMSD $<1.5 \AA$ for coordinates; $\mathrm{TM}>0.6$ to a representative structure for fold level prompts; and SS3 accuracy $>80 \%$ for secondary structure prompts) with high confidence ( $\mathrm{pTM}$ $>0.8$, pLDDT $>0.8$ ). Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - Motifs and scaffolds are combined with specific fold architecture prompts.
We find that ESM3 is able to solve a wide variety of such tasks (Fig. 2C). It does so without retrieving the motif's original scaffold (median TM-score of $0.40 \pm 0.10$ to reference protein; Appendix A.3.9). In some cases, the scaffolds are transferred from existing proteins which have similar motifs (for example, the ESM3-designed alpha-helical scaffold for the zinc-binding motif has high similarity to $\mathrm{Ni}_{2+}$-binding proteins, PDB: 5DQW, 5DQY; Fig. 2C, row 3 column 1). For many motifs (e.g., binding sites for serotonin, calcium, protease inhibitor, and Mcl-1 inhibitor) Foldseek (39) finds no significant similarity to other proteins that contain the same motif. In these cases we observe that sometimes the motif has been grafted into entirely different folds (e.g. a protease inhibitor binding site motif in a beta-barrel which is most similar to a membrane-bound copper transporter, PDB: 7PGE; Fig. 2C, row 3 column 3). At other times, the scaffold appears to be entirely novel, such as an alpha/beta protein designed to scaffold the Mcl-1 inhibitor binding motif, which has low structural similarity to all known proteins in the PDB, ESMAtlas, and the AlphaFold databases (max. TM-score $<0.5$; Fig. 2C, row 4 column 1). Overall, the generated solutions have high designability, i.e. confident recovery of the original structure after inverse folding with ESMFold (median pTM $0.80 \pm 0.08$; scTM $0.96 \pm 0.04$; Appendix A.3.9). Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - ESM3 can solve a wide variety of tasks.
Through experiments with prompt engineering, we have observed especially creative responses to prompts. Here, we highlight an example of protein compression. Starting from a natural trypsin (PDB $1 \mathrm{Y} 3 \mathrm{~V}$ ), we prompt with the sequence and coordinates of the catalytic triad as well as functional keywords describing trypsin, but reduce the overall generation length by a third (from 223 to 150 residues). ESM3 maintains the coordination of the active site (cRMSD $0.73 \AA$ ) and the overall fold with high designability (pTM 0.84 , scTM mean 0.97 , std 0.006), despite the significant reduction in sequence length and the fold only being specified by the function keyword prompt (Fig. 2D). Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - Experiments with prompt engineering have resulted in particularly creative responses.
These examples illustrate ESM3's ability to find creative solutions to prompts specified in any of its input tracks, individually or in combination. This capability enables a rational approach to protein design, providing control at various levels of abstraction, from high-level topology to atomic coordinates, using a generative model to bridge the gap between the prompt and biological complexity.
Please analyze this paragraph from the point of view of an expert scientist and be critical of any omissions or possible sources of problems you see.
AI: The description of the innovative capabilities of ESM3 in solving a variety of protein design tasks using prompt engineering is impressive, yet there are several areas where the information provided could be critiqued for completeness, clarity, or potential for overlooking critical factors:
Validation Methods and Statistical Significance: While the paragraph mentions a variety of specific metrics (cRMSD, TM-score, SS3 accuracy, pTM, and pLDDT) to validate the generated protein structures, it lacks a detailed explanation of the statistical methods used to establish these thresholds. Moreover, it does not discuss the statistical significance of the results, which is critical for understanding the reliability and generality of the findings across the protein design landscape.
Experimental Verification: The document leans heavily on computational assessments like cRMSD and TM-scores but does not mention any biochemical or biophysical experimental validation of the designed proteins. Considering the complex nature of protein folding and function, computational predictions, no matter how sophisticated, cannot fully replace empirical observations. Information on the stability, solubility, or functional activity of the designed proteins would enhance the credibility of the claims.
Comparison with Existing Approaches: While the capability of ESM3 is emphasized, there is inadequate context regarding how this approach compares to existing methods in protein design. Mention of specific advantages over or complementarities with methods like AlphaFold, Rosetta, or others would provide a clearer picture of ESM3's place within the broader field.
Potential for Bias or Overfitting: ESM3's ability to generate protein designs without retrieving the motif's original scaffold is highlighted, but there is insufficient discussion about the potential for overfitting or bias in the learning model. For instance, whether ESM3 could be favoring certain protein motifs or folds over others based on the training data's composition would be worth addressing.
Limitation in Diversity of Protein Structures: Although ESM3 designed proteins with motifs grafted into entirely different folds is intriguing, the paragraph does not disclose any limitations encountered due to the immense diversity of protein structures. It would be beneficial to know if there were scenarios where ESM3 struggled to generate satisfactory designs or if any patterns were less amenable to design within its computational framework.
Technical Specifications and Accessibility: Details on the computational resources required for ESM3, the learning models' specifics, and the degree of expertise needed to operate ESM3 would be valuable. Additionally, information on the accessibility of ESM3 for the wider scientific community could foster further research and application.
Ethical Considerations and Risks: The paragraph omits any discussion on the ethical considerations and potential risks associated with rapidly advancing protein design technologies. Issues of biosecurity, dual-use research, and ecological implications of introducing novel proteins into living systems warrant consideration and open discussion.
Addressing these points could strengthen the argument, provide a more rounded view of ESM3's capabilities and limitations, and foster a deeper understanding of its potential impact on the field of protein design.
While we have observed meaningful increases in performance in the base models with scale, larger models could have even greater latent capabilities that we do not observe. The base ESM3 models can be prompted to perform difficult tasks such as atomic coordination and composition of prompts, despite the fact that the models have not been explicitly optimized for these objectives. Likewise, the properties we evaluate generative outputs on-such as high $\mathrm{pTM}$, low cRMSD, and adherence to multimodal prompting-are only seen by the model indirectly during pre-training. Aligning the model directly to these tasks with finetuning could elicit even greater capability differences with larger models. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - The base models have shown meaningful increases in performance with scale.
We study how the base models can be aligned (40) to generate proteins that satisfy challenging prompts. To do this, for each model we construct a dataset of partial structure prompts, generate multiple protein sequences for each prompt, and then fold and score each of the sequences using ESM3 for consistency with the prompt (cRMSD) and foldability (pTM). High quality samples are paired with low quality samples for the same prompt to construct a preference dataset (Appendix A.4). ESM3 is then tuned to optimize a preference tuning loss, which incentivizes the model to put higher likelihood on the high quality samples compared to low quality samples (Appendix A.4) (41, 42). Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - Base models are aligned to generate proteins that meet challenging prompts.
After aligning the ESM3 1.4B, 7B, and 98B base models, we evaluate their absolute performance, and the shift in the distribution of generations. To measure consistency of a generation with a prompt, the generated sequence is folded and success is measured based on structural metrics (backbone cRMSD $<1.5 \AA$ ) and foldability (pTM $>0.8$ ). To ensure that the model used for evaluation is orthogonal to that used for creating the preference dataset, we conduct these evaluations using ESMFold. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - ESM3 1.4B, 7B, and 98B base models are aligned for evaluation.
We examine the ability of the model to generate highquality scaffolds using challenging tertiary motif scaffolding prompts. We prompt ESM3 with the amino acid identities and atomic coordinates of residues derived from a dataset of 46 ligand binding motifs in a set of temporally held out proteins (Appendix A.4.5). For each motif task, we create 1024 prompts by permuting the order of the residues, varying their position in the sequence, and varying the length of the sequence. A single protein is generated per prompt. We evaluate success using the percentage of tasks solved (backbone cRMSD $<1.5 \AA$, pTM $>0.8$ ) after 128 generations (Appendix A.4.5). Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - The model's capability to generate high-quality scaffolds is being examined through challenging tertiary motif scaffolding prompts.
Preference tuned models solve double the atomic coordination tasks compared to base models (Fig. 3A). While the base models show differences in the fraction of tasks solved $(9.5 \%$ for 1.4B, $19.0 \%$ for 7B, 26.8\% for 98B; Fig. 3A), a much larger capability difference is revealed through align- Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - Preference tuned models solve double the atomic coordination tasks compared to base models.
ment $(9.5 \%$ to $18.8 \%, 19.0 \%$ to $37.4 \%, 26.8 \%$ to $65.5 \%$ for the 1.4B, 7B and 98B models, respectively). Preferencetuned models not only solve a greater proportion of tasks, but also find a greater number of solutions per task, as evaluated by the number of distinct structural clusters ( $\mathrm{TM}>0.8$ ) with backbone cRMSD $<1.5$ Åand pTM $>0.8$ (Fig. 3B). A shift in the distribution of ESMFold pTM and backbone cRMSD for each ligand binding motif is observed (Fig. 3C; Fig. S17). At the 98B scale, the finetuned model produces more distinct successful clusters than the base model on 37 of the 46 tested ligands, while the remaining 9 ligands were not solved by either the base or aligned model, indicating that alignment almost universally improves the faithfulness to the prompt and the foldability of the generated proteins. Compared to a supervised finetuning baseline, which only maximizes the likelihood of the positive examples, preference tuning leads to larger improvements at all scales (Appendix A.4.6).
These results demonstrate that preference tuning extracts latent capability in the models. The capability of larger models to solve challenging tasks become far more apparent after alignment. Since alignment can be performed with arbitrary objectives, this is an indication of a general ability to respond to finetuning that greatly improves with scale.
Figure 3. The ability to solve complex tasks increases with scale through alignment. ESM3 is aligned to follow atomic coordination prompts with a dataset of preference pairs constructed from prompted generations, where positive samples with good scores for desired properties (high pTM, low cRMSD) are paired with negative samples with worse scores. The preference tuning loss encourages the model to put higher likelihood on the positive samples. After training, models are evaluated by prompting with coordinates in tertiary contact. (A) We show the effect of finetuning on the fraction of tasks solved with 128 generations (Pass@ 128). A large gap opens between the models with scale. The response to alignment shows a latent capability to solve complex tasks in the largest model. Error bars show 2 standard deviations. (B) Number of distinct solutions (clustered at $\mathrm{TM}>0.8$ ) generated for each tertiary motif. After finetuning we often see a number of unique structures for ligands for which we have successes. (C) Densities of prompted generations are shown for the base model (left) and aligned model (right) at the 98B scale for a number of randomly selected ligands. After alignment, the fidelity to the prompt (cRMSD) and quality of generations (pTM) tends to improve meaningfully. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - Figure 3 focuses on how scalability impacts the ability to solve complex tasks through alignment.
AI: Analyzing the provided paragraph critically, several possible sources of problems or omissions emerge, relating to the methodology, evaluation metrics, and generalization of the findings. Here is a breakdown of each fact or idea presented in the paragraph, with attention to areas that might merit further scrutiny or clarification:
Alignment to Atomic Coordination Prompts
ESM3 is aligned using a specific dataset composed of preference pairs created from prompted generations.
Positive samples have high protein tertiary structure modeling (pTM) scores and low cumulative Root Mean Square Deviation (cRMSD) scores, indicative of desired properties.
Negative samples have worse scores in these metrics.
The preference tuning loss function encourages the model to assign a higher likelihood to positive samples.
Evaluation of Models
Models are evaluated based on their performance when prompted with coordinates in tertiary contact.
The primary metric is the fraction of tasks solved with 128 generations (Pass@128).
Effect of Fine-tuning
Fine-tuning significantly affects the ability of models to solve tasks, especially at larger scales.
A notable performance gap exists between models of different scales after fine-tuning.
This suggests a latent capability in larger models to solve complex tasks post alignment.
Error bars in the analysis represent two standard deviations, indicating the variability in the results.
Diversity of Solutions
The number of distinct solutions (clusters with TM>0.8) is measured for each tertiary motif.
Post-fine-tuning, many unique structures are observed for successful ligands.
Quality of Generations
Densities of prompted generations before and after alignment are compared for the base model and aligned model at the 98B scale.
Alignment improves fidelity to the prompt (as measured by cRMSD) and the quality of the generations (as indicated by pTM).
Critical Observations:
Each point highlights a different aspect of the study outlined in the paragraph, reflecting both its contributions and areas where further clarification or investigation would strengthen the findings.
AI: The paragraph presents compelling evidence that preference tuning can significantly enhance the model's ability to generate accurate protein structures in response to complex prompts. However, several critical areas require further scrutiny or clarification for a comprehensive understanding and assessment of the reported results. From an expert scientific perspective, the following points highlight potential sources of concern, omissions, or areas for improvement:
Generalizability of Findings: The paragraph focuses on the improvement of model performance in generating protein structures based on atomic coordination tasks. While impressive, it's crucial to question how well these findings generalize to other types of biological modeling tasks. Does preference tuning offer similar enhancements across various scales of protein modeling challenges, such as predicting protein-protein interactions, allosteric sites, or functional annotations?
Methodological Details: While the paragraph mentions the construction of a dataset for preference tuning and the use of specific metrics like pTM, cRMSD, and the generation of preference pairs, it lacks detailed methodological transparency. For instance, how were the "high quality" and "low quality" samples distinguished and validated as such, beyond the described metrics? The scientific rigor demands clarity on these definitions and validation processes.
Bias and Representativeness: The selection of ligand binding motifs from a set of temporally held-out proteins raises questions about the dataset's representativeness. How were these motifs selected, and do they cover a diverse range of protein functions, structures, and complexities? Bias in dataset selection can significantly impact the model’s perceived performance and its real-world applicability.
Comparative Analysis: It's mentioned that preference tuning leads to larger improvements at all scales compared to a supervised fine-tuning baseline. However, details about the comparative models, their scales, and specific results are lacking. A more detailed comparative analysis, possibly including other state-of-the-art models or techniques, would provide a clearer understanding of the preference tuning's unique contributions.
Error Analysis and Model Limits: The acknowledgment of tasks and ligands not solved by either the base or aligned models is positive. However, a more in-depth error analysis would greatly benefit the scientific narrative. Understanding why certain ligands remain unsolved and identifying the limitations of the current model are essential for guiding future improvements and research directions.
Scalability and Efficiency: While it's noted that larger models show significant improvements after alignment, the computational cost and efficiency of scaling and fine-tuning these models are not addressed. An analysis of the trade-offs between scale, accuracy, and computational resources would be invaluable, especially considering the environmental and economic costs associated with training large-scale models.
Real-world Applicability and Clinical Relevance: Ultimately, the true test of these models lies in their applicability to real-world biological problems and clinical challenges. The paragraph does not discuss how these generated protein structures could be validated experimentally or their potential impact on drug discovery, protein engineering, or understanding disease mechanisms.
In conclusion, while the results presented are promising and highlight the potential of preference tuning in improving the models' ability to generate accurate protein structures, a more in-depth analysis addressing the above points is crucial for assessing the true scientific value and applicability of the findings.
We sought to understand if the base pre-trained ESM3 model has sufficient biological fidelity to generate functional proteins. We set out to create a functional green fluorescent protein (GFP) with low sequence similarity to existing ones. We chose the functionality of fluorescence because it is difficult to achieve, easy to measure, and one of the most beautiful mechanisms in nature. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - An intent to determine the capability of the pre-trained ESM3 model in generating functional proteins was established.
Please comment on the use of GFP as the model system here. Please focus on any possible issues with using this system as a self-reporter, rather than as a reporter as it is usually used. Please comment on the fact that it can take days for the flurophore to appear. Please act as a critical reviewer for a scientific paper. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: Certainly, acting as a critical reviewer, I will dissect the concerns about using GFP (Green Fluorescent Protein) as a self-reporter rather than its conventional use as a reporter. Here are the critical comments broken down into key points:
GFP is traditionally used as a reporter gene allowing researchers to track gene expression, protein localization, and intracellular processes in live cells, owing to its bright green fluorescence when exposed to ultraviolet light.
Utilizing GFP as a self-reporter shifts its application from indicating the presence or activity of other proteins or processes to reporting on its own expression and folding state.
This shift in application could lead to inherent issues:
Sensitivity to Environmental Conditions: GFP fluorescence is susceptible to the cellular environment, including pH and ion concentrations. As a self-reporter, any changes in its fluorescence might not solely reflect changes in the targeted processes but could also result from alterations in the cellular environment.
Maturation Time: One significant drawback of using GFP in this new context is the lag it exhibits before fluorescence becomes visible. This maturation can take hours to days, which might not be suitable for tracking rapid cellular events or processes. In real-time experiments or studies requiring immediate feedback, this delay could lead to misinterpretation or missed observations.
Stability Issues: While GFP is generally stable, varying conditions within different cells or organisms can affect its stability. As a self-reporter, its stability could directly influence the accuracy and reliability of the data collected, potentially leading to variability that complicates data interpretation.
Auto-fluorescence Interference: Many organisms and cells inherently possess substances that can fluoresce, which might interfere with the detection of GFP fluorescence. When used as a self-reporter, distinguishing GFP fluorescence from background auto-fluorescence becomes critical, requiring more stringent controls and possibly complicating data analysis.
Quantitative Limitations: Quantifying GFP fluorescence as a measure of its self-reporting capacity could be challenging. The fluorescence intensity might not linearly correlate with the quantity of GFP or its functional state, making it difficult to draw quantitative conclusions without well-designed calibration experiments.
In summary, while GFP's unique properties have revolutionized biological research, repurposing it from a reporter of other molecular phenomena to a self-reporter introduces complexities that necessitate careful experimental design, controls, and interpretations. Adjustments or additional methodologies may be required to mitigate these issues, ensuring accurate and meaningful data.
AI: The selection of green fluorescent protein (GFP) as the model system to test the biological fidelity of the base pre-trained ESM3 model's ability to generate functional proteins with low sequence similarity to existing proteins is commendable for several reasons. Nonetheless, there are pertinent issues worth discussing, especially regarding its use as a self-reporter rather than as a traditional reporter, and the inherent delay in fluorophore maturation:
GFP as a Choice for Testing Biological Fidelity: The choice of GFP is logical due to its straightforward, quantifiable readout of fluorescence, facilitating easy assessment of the model's success in generating functional proteins. Its broad use in biological research for tracking gene expression and protein localization further underscores its utility. However, using GFP, a well-characterized and widely studied protein, might have limitations in truly gauging the model's novel protein-generating capabilities, especially since GFP's folding and fluorescence mechanisms are relatively well understood and might not challenge the model's capabilities to the fullest extent possible with more complex proteins or less characterized functionalities.
GFP as a Self-Reporter: Using GFP as a self-reporter rather than its traditional role as a tag for other proteins presents unique challenges. Typically, GFP is fused to another protein of interest, allowing researchers to track the expression, localization, and dynamics of the tagged protein. However, when used as a self-reporter, the direct fluorescence measurement pertains solely to the presence and proper folding of GFP itself, without providing insight into the functionality or behavior of other protein systems. This approach might limit the scope of inference regarding the model's ability to predict or design functional domains critical for complex protein-protein interactions or other functional aspects beyond mere fluorescence.
Delayed Fluorophore Maturation: An inherent challenge with GFP is the time required for the fluorophore to mature, which can take several days. This delay can introduce several variables into the experiment, including but not limited to, varying expression levels over time, potential degradation of misfolded proteins, and changes in cellular conditions that might affect the fluorescence intensity independently of the protein's initial folding and functionality. Such delays can complicate the interpretation of results, as the fluorescence might not accurately reflect the immediate success of the protein design but rather the culmination of several days of cellular processes. Additionally, it raises concerns about the efficiency and throughput of using this system for large-scale testing of newly designed proteins, which could significantly slow down the experimental cycle.
Critical Evaluation and Recommendations: In conclusion, while using GFP as a model system has its merits, the authors should consider augmenting their study with additional model systems that can provide immediate feedback on functionality and reduce the influence of delayed maturation on the experimental outcomes. Exploring the use of rapidly maturing fluorescent proteins or alternative functional assays that allow for real-time monitoring of protein activity could enhance the robustness and interpretability of the results. Furthermore, incorporating parallel strategies to evaluate the ESM3 model's capabilities in designing proteins with more complex functionalities than fluorescence could provide a more comprehensive understanding of its biological fidelity and predictive power.
Addressing these considerations will strengthen the authors' conclusions and provide deeper insights into the capabilities and limitations of pre-trained models like ESM3 in generating functionally diverse and novel proteins.
ME: Specifically, I am interested in the fact that the protein autocatalyzes to make the flurophore and that if this happens over a span of a few days, then it is a red flag.
AI: The autocatalytic formation of the fluorophore within green fluorescent protein (GFP) is indeed a critical component that influences its utility in this experimental context. GFP contains a sequence that undergoes a series of spontaneous reactions, without the aid of additional enzymes, to form the chromophore that emits visible green light upon excitation. This process, while fascinating for its self-sufficiency and efficiency in nature, introduces specific challenges in a setting where the aim is to evaluate the functionality of newly generated proteins based on their sequence similarity or dissimilarity to existing ones:
Delayed Functional Assessment: The fact that the autocatalytic formation of GFP’s fluorophore can take days is a significant limitation for rapid throughput screening or experimental cycles. This delay makes it difficult to quickly assess whether a novel protein variant generated by the ESM3 model is functional, as you must wait for the chromophore to mature to confirm fluorescence. In a dynamic research environment where time is often of the essence, such delays could be a bottleneck.
Variable Efficiency of Chromophore Formation: Different GFP variants might have variations in the efficiency of fluorophore formation. If the model generates a GFP variant with lower sequence similarity to known GFPs, this could potentially affect the autocatalysis kinetics, leading to even longer maturation times or incomplete chromophore formation, thereby affecting fluorescence intensity and misleading interpretation regarding the functionality of the protein.
Misinterpretation of Results: The time-dependent nature of GFP fluorescence development means that early assessments of fluorescence could yield negative results, not because the protein is non-functional but simply because the chromophore has not yet matured. This could lead to the incorrect classification of potentially functional proteins as failures.
Impact on Experimental Design and Interpretation: Researchers must design their experiments to account for the delayed maturation, likely incorporating longer observation periods or additional controls to determine when the fluorescence has reached its peak. This makes experimental design and result interpretation more complex and may introduce confounding variables, such as the stability of the protein over the extended maturation period or the effects of cellular conditions on protein expression levels.
Time-Resolved Measurements: Implementing time-resolved measurements to monitor fluorescence development over several days could help in accurately capturing the point of maturation and functional onset of the newly synthesized GFP variants.
Alternative Rapid Maturation Reporters: Exploring the use of alternative fluorescent proteins or reporters that mature more quickly than GFP could serve as a complementary approach for faster functional assessments, although it may introduce the challenge of transferring findings between different reporter systems.
Molecular Engineering: Considering molecular engineering strategies to modify GFP or create GFP-like proteins with accelerated chromophore maturation could help mitigate some of these issues. Such strategies might involve designing mutations known to affect the rate of chromophore formation.
Comprehensive Analysis and Controls: Including a comprehensive set of analytical and control measures to account for potential delays and variability in chromophore formation. This might involve parallel experiments with known GFP variants to provide benchmarks or baselines for comparison.
In summary, while GFP’s autocatalytic formation of its fluorophore is a unique feature that has made it an invaluable tool in molecular biology, its application in evaluating the functional generation of proteins by the ESM3 model introduces specific challenges that require careful experimental design and interpretation. Addressing these issues adequately can ensure that the findings are robust and provide a true reflection of the model’s capabilities.
Responsible for the fluorescence of jellyfish and the vivid colors of coral (43), proteins in the GFP family are unique in their ability to form a fluorescent chromophore without cofactors or substrates (27). This property allows the GFP sequence to be inserted into the genomes of other organisms to visibly label molecules, cellular structures, or processes, providing a foundational toolkit that has been broadly applied across the biosciences. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - Proteins in the GFP family are responsible for the fluorescence of jellyfish and the vivid colors of coral.
The GFP family has been the subject of decades of protein engineering efforts, but still the vast majority of functional variants have come from prospecting the natural world. Rational design and machine learning-assisted highthroughput screening have yielded GFP sequences with improved properties-such as higher brightness or stability, or differently colored variants-that incorporated small numbers of mutations (typically 5 to 15 , out of the total 238 amino acid coding sequence) from the originating sequence. Studies have shown that only a few random mutations reduces fluorescence to zero (44-46). whereas in rare cases, leveraging high throughput experimentation, scientists have been able to introduce up to $40-50$ mutations i.e. a $20 \%$ difference in total sequence identity $(44,47,48)$ while retaining GFP fluorescence. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - The GFP family has been extensively engineered for decades.
Generating a new GFP would require materialization of the complex biochemistry and physics that underlie its fluorescence. In all GFPs, an autocatalytic process forms the chromophore from three key amino acids in the core of the protein. The unique structure of GFP, a kinked central alpha helix surrounded by an eleven stranded beta barrel with inward facing coordinating residues, enables this reaction (49). Once formed, the chromophore must not just absorb light but also emit it in order to be fluorescent. Light emission is highly sensitive to the local electronic environment of the chromophore. For these reasons, obtaining a new functional GFP would require precise configuration of both the active site and the surrounding long range tertiary interactions throughout the beta barrel.
Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - Generating a new GFP involves complex biochemistry and physics related to its fluorescence.
AI: - Generating a new GFP involves the complex biochemistry and physics of fluorescence.
Figure 4. Generating a new fluorescent protein with a chain of thought. (A) We prompt ESM3 with the sequence and structure of residues required for forming and catalyzing the chromophore reaction, as well as the structure of part of the central alpha helix from a natural fluorescent protein (left). Through a chain of thought, ESM3 generates design candidates (right). (B) ESM3 found a bright GFP distant from other known GFPs in two experiments. We measured fluorescence in E. coli lysate. Top row, photograph of plates. Bottom row, plate reader fluorescence quantification. Positive controls of known GFPs are marked with purple circles, negative controls with no GFP sequence or no E. Coli are marked with red circles. In the first experiment (left) we expressed designs with a range of sequence identities. A notable design with low sequence identity to known fluorescent proteins appears in the well labeled B8 (highlighted in a black circle bottom, white circle top). We continue the chain of thought from the protein in B8 for the second experiment (right). A bright design appears in the well labeled C10 (black circle bottom, white circle top) which we designate esmGFP. (C) esmGFP exhibits fluorescence intensity similar to common GFPs. Normalized fluorescence is shown for a subset of proteins in experiment 2. (D) Excitation and emission spectra for esmGFP overlaid on the spectra of EGFP. (E) Two cutout views of the central alpha helix and the inside of the beta barrel of a predicted structure of esmGFP. The 96 mutations esmGFP has relative to its nearest neighbor, tagRFP, are shown in blue. (F) Cumulative density of sequence identity between fluorescent proteins across taxa. esmGFP has the level of similarity to all other FPs that is typically found when comparing sequences across orders, but within the same class. (G) Evolutionary distance by time in millions of years (MY) and sequence identities for three example anthozoa GFPs and esmGFP. (H) Estimator of evolutionary distance by time (MY) from GFP sequence identity. We estimate esmGFP is over 500 million years of natural evolution removed from the closest known protein. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - (A) ESM3 is prompted with the sequence and structure of residues essential for forming and catalyzing the chromophore reaction and the structure of part of a natural fluorescent protein's central alpha helix to generate design candidates.
In an effort to generate new GFP sequences, we directly prompt the base pretrained 7B parameter ESM3 to generate a 229 residue protein conditioned on the positions Thr62, Thr65, Tyr66, Gly67, Arg96, Glu222, which are critical residues for forming and catalyzing the chromophore reaction (Fig. 4A). We additionally condition on the structure of residues 58 through 71 from the experimental structure in 1QY3, which are known to be structurally important for the energetic favorability of chromophore formation (50). Specifically, sequence tokens, structure tokens, and atomic coordinates of the backbone are provided at the input and generation begins from a nearly completely masked array of tokens corresponding to 229 residues, except for the token positions used for conditioning. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - A base pretrained 7B parameter ESM3 model is used to generate new GFP sequences.
We generate designs using a chain-of-thought procedure as follows. The model first generates structure tokens, effectively creating a protein backbone. Backbones that have sufficiently good atomic coordination of the active site but differentiated overall structure from the 1QY3 backbone pass through a filter to the next step of the chain. We add the generated structure to the original prompt to generate a sequence conditioned on the new prompt. We then perform an iterative joint optimization, alternating between optimizing the sequence and the structure. We reject chainsof-thought that lose atomic coordination of the active site (Appendix A.5.1). We draw a computational pool of $10 \mathrm{~s}$ of thousands of candidate GFP designs from the intermediate and final points in the iterative joint optimization stage of the generation protocol. We then bucket the designs by sequence similarity to known fluorescent proteins and filter and rank designs using a variety of metrics (details in Appendix A.5.1.5) Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - The process begins with generating structure tokens to create a protein backbone.
We performed a first experiment with 88 designs on a 96 well plate, with the top generations in each sequence similarity bucket. Each generated protein was synthesized, expressed in E. coli, and measured for fluorescence activity at an excitation wavelength of $485 \mathrm{~nm}$ Fig. 4B left. We measured brightness similar to positive controls from a number of designs that have higher sequence identity with naturally occurring GFPs. We also identify a design in well B8 (highlighted in a black circle) with only $36 \%$ sequence identity to the 1QY3 sequence and $57 \%$ sequence identity to the nearest existing fluorescent protein, tagRFP. This design was 50x less bright than natural GFPs and its chromophore matured over the course of a week, instead of in under a day, but it presents a signal of function in a new portion of sequence space that to our knowledge has not been found in nature or through protein engineering. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - An experiment was conducted with 88 designs on a 96 well plate to test generated proteins.
We continue the chain of thought starting from the sequence of the design in well B8 to generate a protein with improved brightness, using the same iterative joint optimization and ranking procedure as above. We create a second 96 well plate of designs, and using the same plate reader assay we find that several designs in this cohort have a brightness in the range of GFPs found in nature. The best design, located in well C10 of the second plate (Fig. 4B right), we designate esmGFP. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - A second 96 well plate of designs was created to improve protein brightness, continuing from an initial design in well B8.
We find esmGFP exhibits brightness in the distribution of natural GFPs. We evaluated the fluorescence intensity at 0 , 2 , and 7 days of chromophore maturation, and plot these measurements for esmGFP, a replicate of B8, a chromophore knockout of B8, along with three natural GFPs avGFP, cgreGFP, ppluGFP (Fig. 4C). esmGFP takes longer to mature than the known GFPs that we measured, but achieves a comparable brightness after two days. To validate that fluorescence was mediated by the intended Thr65 and Tyr66, we show that B8 and esmGFP variants where these residues were mutated to glycine lost fluorescence activity (Fig. S21). Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - EsmGFP exhibits brightness in the distribution of natural GFPs.
ME: Please comment on the length of time of maturation of the chromophore compared to other experiments of this type.
AI: The paragraph indicates that esmGFP takes longer to mature than the known GFPs measured in the experiment, maturing to a comparable brightness after two days. This comparison implies that the pace of chromophore maturation in other experiments with GFPs typically might be faster than what is observed for esmGFP. In the realm of GFP research, the maturation rate of the chromophore is a critical factor because it can influence the utility of GFPs in various experimental setups, particularly those requiring rapid tagging and visualization of proteins. Faster-maturing variants are often preferred for real-time or dynamic studies where the rapid availability of the fluorescent signal is crucial.
However, it's also noteworthy that while esmGFP has a slower initial maturation time, it ultimately achieves a brightness comparable to natural GFPs after two days. This suggests that for applications where the timing of initial fluorescence is less critical, esmGFP could still be a viable option. The prolonged maturation time might be a drawback for certain time-sensitive applications but might not significantly impact studies where the ultimate brightness, rather than the speed of fluorescence onset, is the primary concern.
Comparatively, in other experiments involving GFPs or similar fluorescent proteins, the selection of the specific variant often hinges on the balance between the speed of chromophore formation and other properties like photostability, brightness, and spectral characteristics. The relatively slower maturation of esmGFP could, therefore, be seen as a limitation in the context of these experiments, especially when contrasted with other GFP variants known for quicker chromophore maturation.
Analysis of the excitation and emission spectra of esmGFP reveals that its peak excitation occurs at $496 \mathrm{~nm}$, which is shifted $7 \mathrm{~nm}$ relative to the $489 \mathrm{~nm}$ peak for EGFP, while both proteins emit at a peak of $512 \mathrm{~nm}$ (Fig. 4D). The shapes of the spectra indicated a narrower full-widthhalf-maximum (FWHM) for the excitation spectrum of esmGFP (39mm for esmGFP vs $56 \mathrm{~nm}$ for EGFP), whereas the FWHM of their emission spectra were highly comparable ( $35 \mathrm{~nm}$ and $39 \mathrm{~nm}$, respectively). Overall esmGFP exhibits spectral properties consistent with known GFPs. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - Peak excitation of esmGFP is at 496 nm.
Please analyze this experiment critically, like a reviewer for Nature or Science would review a paper, very critically. Assess the experimental design and results.
AI: The experiment described aims to generate a new green fluorescent protein (GFP) using an innovative combination of computational power, specifically the pretrained ESM3 model, and experimental validation. The objectives and methodology represent a cutting-edge approach to synthetic biology and protein engineering. However, to critically assess the work similar to a reviewer for high-caliber journals like Nature or Science, several aspects should be meticulously evaluated, including the clarity of innovation, experimental design, data analysis, and the broader implications. Below are some critical observations and questions that would likely be raised during the peer-review process:
In summary, while the experiment represents an impressive fusion of computational biology and experimental validation, the manuscript would benefit from a deeper and more detailed exploration of the methods, a clearer articulation of the novelty and implications of the findings, and a thorough statistical and functional analysis of the results. Addressing these aspects would significantly strengthen the manuscript and highlight its contribution to the field of protein engineering and beyond.
ME: The protein that they find has 58% sequence identity, didn't they just find an existing fold that was distant from their existing protein?
AI: The discovery of a protein with 58% sequence identity to the nearest existing fluorescent protein, such as tagRFP, in the context of protein engineering, indeed suggests the identification of a protein that shares some similarity with known proteins yet is significantly distinct in its sequence to warrant interest as a novel entity. This level of identity indicates that while there might be a common structural fold or functional motif shared between esmGFP and its distant relatives (like tagRFP), the variations present are sufficient to classify it as a new design or variant within the broader family of fluorescent proteins.
In protein science, the notion of "sequence identity" is crucial for understanding relationships between proteins. High sequence identity (>70%) often implies similar structure and function, while lower sequence identity, particularly in the range of 30% to 70%, can still indicate a shared fold but suggests significant functional or regulatory divergence. Here are a few points to consider regarding the discovery:
Existing Fold with Modifications: It's common in protein engineering to create variants that share an existing fold - the core three-dimensional structure - but possess significant differences in sequence that confer new properties or functions. The fold provides a stable framework, while sequence variations can alter properties such as fluorescence wavelength, intensity, stability, and chromophore formation dynamics.
Novelty and Functionality: A protein with 58% sequence identity to known fluorescent proteins, exhibiting distinct fluorescence characteristics (i.e., different excitation/emission wavelengths, brightness, or chromophore maturation kinetics), represents a functional novelty. This is particularly true if these properties are advantageous or desirable for specific applications (e.g., bioimaging in contexts where existing GFPs are suboptimal).
Engineered vs. Natural Variants: In the pursuit of new fluorescent proteins, both entirely novel discoveries from nature and engineered variants of known proteins are valuable. In this case, using ESM3 to generate a protein significantly distant in sequence space from known GFPs, yet retaining or even improving upon the desired fluorescent functionality, underscores the potential of computational methods in expanding the toolkit of useful fluorescent proteins beyond what nature has offered.
Broader Implications: Finding or designing proteins with moderate sequence identity to known entities but with unique features highlights the plasticity of protein structures and their functional adaptability. It underscores the principle that relatively small changes in sequence (reflected in less than absolute sequence identity) can lead to significant functional diversification. This principle is fundamental in evolutionary biology and protein engineering.
In summary, the identification of a protein with 58% sequence identity to known fluorescent proteins is significant because it represents a balance between utilizing known structural motifs for stability and introducing sufficient novelty to achieve distinct functional properties. This approach is at the heart of protein engineering and synthetic biology, leveraging computational tools to explore the vast sequence space for functional innovation.
ME: How much evolutionary time would pass normally for 58% sequence identity?
AI: Inferring the amount of evolutionary time that corresponds to a certain level of sequence identity between proteins is a complex task, as the rate at which proteins evolve can vary widely among different families of proteins, different organisms, and even within different regions of the same protein. However, we can provide some general context and approximate estimations based on known principles of molecular evolution.
The concept of a "molecular clock" proposes that the rate of evolutionary change in some molecules (often proteins or DNA) is approximately constant over time, allowing the time since two species diverged to be estimated by measuring the genetic distance between them. However, this rate is not universal and can be influenced by factors like the functional constraints on the protein, the organism's lifespan, population size, and its overall mutation rate.
High Variability in Rates: The evolutionary rate can vary significantly between different proteins. For example, proteins involved in basic cellular processes and with critical functions tend to evolve slowly (low mutation rate), while proteins involved in interactions with other species (like immune system proteins or toxins) may evolve much faster.
General Rule of Thumb: For many proteins, a rough approximation is that a 1% sequence divergence corresponds to about 1-2 million years of evolution, though this figure can range widely depending on the specific evolutionary pressures acting on the protein.
Calculating Time for 58% Sequence Identity: If we had a hypothetical baseline where two proteins are 100% identical, and we assume a conservative estimate that 1% divergence corresponds to 1.5 million years, a 42% sequence divergence (100% - 58%) would suggest an evolutionary divergence time of approximately 63 million years (42 * 1.5 million years). It's important to note, however, that this calculation assumes a constant rate of evolution, which is often not the case in reality.
For the specific case of esmGFP and its 58% sequence identity to the nearest known fluorescent protein:
In conclusion, while calculating evolutionary time based on sequence identity provides an interesting perspective on the novelty and potential evolutionary distance of proteins like esmGFP, it's important to approach these estimates with caution, acknowledging the complexities and variabilities involved.
We next sought to understand how the sequence and structure of esmGFP compares to known proteins. A BLAST (51) search against the non-redundant protein sequences database and an MMseqs (52) search of ESM3's training set report the same top hit-tagRFP, which was also the nearest neighbor to B8-with $58 \%$ sequence identity, representing 96 mutations throughout the sequence. tagRFP is a designed variant, and the closest wildtype sequence to esmGFP from the natural world is eqFP578, a red fluorescent protein, which differs from esmGFP by 107 sequence positions ( $53 \%$ identity). Sequence differences between esmGFP and tagRFP occur throughout the structure (Fig. 4E) with 22 mutations occurring in the protein's interior, which is known to be intensely sensitive to mutations due to chromophore proximity and a high density of interactions (46). Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - A BLAST search against the non-redundant protein sequences database was conducted for esmGFP.
ME: What is tagRFP?
AI: tagRFP is a red fluorescent protein that has been engineered for use in biotechnological and medical applications. It is derived from the green fluorescent protein (GFP) family, which originally comes from jellyfish (specifically Aequorea victoria) and other marine organisms. Unlike the natural GFP, tagRFP has been specifically designed and optimized through molecular biology techniques to emit red light when excited by the appropriate wavelength. Its development allows for its use in various applications, including as a marker in molecular and cellular biology to visualize and track gene expression, protein localization, and intracellular processes in real-time. The engineering process typically involves mutating the original GFP or a related fluorescent protein to shift the emission spectrum from green to red, as well as to improve other properties, such as brightness, photostability, and maturation rate, making it a valuable tool for research and diagnostics.
ME: How similar is tagRFP to GFPs?
AI: tagRFP and GFP (Green Fluorescent Protein) share a fundamental structural similarity but have distinct differences in their spectral properties—that is, the wavelengths of light they absorb and emit.
In conclusion, while tagRFP shares a common structural foundation with GFP, it has been significantly modified to emit red light, making it distinct in terms of spectral properties and applications. These modifications enable the use of both proteins in multi-color labeling experiments in molecular and cellular biology, allowing researchers to track different processes or molecules simultaneously within the same cell or tissue.
ME: Why do they absorb at different wavelengths?
AI: The differing absorption wavelengths of proteins like GFP (Green Fluorescent Protein) and its derivatives, such as tagRFP (a red fluorescent protein), are primarily due to variations in their chromophore structures and the surrounding amino acid environment. Here's a deeper look into why these differences occur:
Chemical Composition: The chromophore is the part of the fluorescent protein responsible for its ability to absorb and emit light. It is formed from a small number of amino acids. In GFP, the chromophore consists of a sequence of three amino acids that undergo a series of post-translational modifications to form a conjugated system capable of absorbing and emitting light. Modifications in the chemical structure of the chromophore, such as variations introduced in red fluorescent proteins (RFPs) like tagRFP, alter the energy levels of the electrons in the system, thereby affecting the wavelengths at which light is absorbed and emitted.
Conjugation and Electron Delocalization: The extent of conjugation and electron delocalization within the chromophore plays a significant role in determining its absorption properties. Increased conjugation, as often seen in RFP chromophores, results in a reduction in the energy gap between the ground state and the excited state, shifting the absorption and emission wavelengths towards the longer, red end of the spectrum.
Amino Acid Surroundings: The specific amino acids surrounding the chromophore influence its photophysical properties through various types of interactions, such as hydrogen bonding, polar interactions, and van der Waals forces. These interactions can alter the electronic properties of the chromophore, further influencing the absorption wavelength.
pH and Ionic Strength: The local environment's pH and ionic strength can affect chromophore ionization and protein folding, thus impacting the chromophore's optical properties.
The understanding of these principles has allowed researchers to expand the fluorescent protein color palette significantly, enabling the development of a variety of fluorescent proteins across the visible spectrum and beyond, which are invaluable tools in biological imaging and molecular biology.
Examination of a sequence alignment of 648 natural and designed GFP-like fluorescent proteins revealed that esmGFP has the level of similarity to all other FPs that is typically found when comparing sequences across taxonomic orders, but within the same taxonomic class (Fig. 4F). For example, the difference of esmGFP to other FPs is similar to level of difference between FPs belonging to the orders of scleractinia (stony corals) and actiniaria (sea anemones) both of which belong to the larger class anthozoa of marine invertebrates (Fig. 4G). The closest FPs to esmGFP come from the anthozoa class (corals and anemones), average sequence identity $51.4 \%$, but esmGFP also shares some sequence identity with FPs from the hydrozoa (jellyfish) where the famous avGFP was discovered, average sequence identity $33.4 \%$ (Fig. S22). Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - Examination involved a sequence alignment of 648 natural and designed GFP-like fluorescent proteins.
We can draw insight from evolutionary biology on the amount of time it would take for a protein with similar sequence identity to arise through natural evolution. In Fig. 4G we show esmGFP alongside three Anthozoan GFPs. We use a recent time-calibrated phylogenetic analysis of the Anthozoans (53) that estimated the millions of years ago (MYA) to last common ancestors to estimate evolutionary time between each pair of these species. Using a larger dataset of six Anthozoan GFPs and species for which we have accurate MYA to last common ancestors and GFP sequence identities, we construct a simple estimator that correlates sequence identity between FPs to MY of evolutionary time between the species (Fig. $4 \mathrm{H}$ ) to calibrate against natural evolution. Based on this analysis we estimate esmGFP represents an equivalent of over 500 million years of evolution from the closest protein that has been found in nature. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - A time-calibrated phylogenetic analysis of Anthozoans was used to estimate the millions of years ago (MYA) to the last common ancestors.
Please review this as if you were a reviewer for Nature Magazine
AI: As a reviewer assessing the analysis and claims made about esmGFP, its comparison to known proteins, and the evolutionary context presented for a publication in Nature, the following points encompass constructive feedback, critical assessment, and recommendations for the authors to consider:
Comprehensive Comparison: The authors have conducted a meticulous comparison of esmGFP against known protein sequences using tools like BLAST and MMseqs, which is commendable. However, the assertion about esmGFP's evolutionary novelty based on the level of sequence identity requires further support. Specifically, it would enhance the manuscript if the authors included a phylogenetic analysis that visually places esmGFP within the context of existing fluorescent proteins (FPs) beyond the top hits (tagRFP and eqFP578).
Structural Implications of Mutations: While the manuscript notes that 22 mutations occur in the protein's interior, near the chromophore, the functional implications of these mutations remain unclear. Do these mutations contribute to stability, fluorescence properties, or other functionally relevant changes? Experimentally derived data, such as changes in quantum yield or maturation time correlated with these mutations, would add valuable insights into the structure-function relationship in esmGFP.
Conservation and Novelty: The manuscript would benefit from a deeper analysis of the conservation of key residues involved in the fluorescence mechanism among FPs. This can help elucidate whether esmGFP’s fluorescence is primarily derived from its sequence novelty or if it adopts a known mechanism with modifications. Additionally, information about solvent accessibility and potential new interactions formed by mutations could further highlight structural novelty.
Methodology for Evolutionary Estimates: The estimation of esmGFP representing over 500 million years of evolution is a profound statement that needs rigorous backing. The methodology for correlating sequence identity with millions of years should be elaborated, including any assumptions and the statistical model used. A critical question is how linear correlation applies across the entire spectrum of sequence identities, especially given evolutionary rates can vary significantly across different proteins and taxa.
Comparative Analysis Across Taxa: The comparison of esmGFP to proteins from both Anthozoans and Hydrozoans provides an interesting evolutionary narrative. Nonetheless, a more detailed analysis comparing the evolutionary pace of fluorescent proteins within these groups versus esmGFP would be enlightening. For instance, how do the evolutionary rates of naturally occurring FPs compare to the rate implied for esmGFP's divergence?
Broader Evolutionary Implications: The claim that esmGFP could represent an equivalent of over 500 million years of natural evolution suggests an evolutionary trajectory that significantly deviates from observed natural processes. The discussion should consider and address potential factors or mechanisms that could allow such a rapid departure in sequence identity within a relatively condensed timeframe, in computational terms. Could horizontal gene transfer, gene duplication, or accelerated mutation rates under selective pressure play a role in a natural counterpart to this artificially accelerated evolution?
Experimental Validation: Beyond sequence analysis, direct experimental evidence demonstrating unique features of esmGFP compared to its closest relatives could strengthen claims of novelty. For example, photostability tests, detailed characterization of fluorescence properties under various conditions, and crystal structure determination can provide concrete evidence of functional novelty.
Clarify Evolutionary Analysis Assumptions: The assumptions underlying the correlation of sequence identity with evolutionary time need clarification. This includes sensitivity analysis to test how changes in these assumptions could affect the estimated 500 million years of evolutionary distance.
Contextualize Within Protein Evolution: The discussion could benefit from a broader context of protein evolution. How does this accelerated "evolution" through computational prediction compare with the fastest examples of evolution observed in nature?
In summary, while the manuscript presents an intriguing concept and preliminary analysis of esmGFP, a more detailed exploration of structure-function relationships, along with a rigorous justification of the evolutionary time estimates and their implications, would significantly enhance the manuscript's impact and credibility.
We have found that language models can reach a design space of proteins that is distant from the space explored by natural evolution, and generate functional proteins that would take evolution hundreds of millions of years to discover. Protein language models do not explicitly work within the physical constraints of evolution, but instead can implicitly construct a model of the multitude of potential paths evolution could have followed. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - Language models can access a protein design space that is significantly distant from the space explored by natural evolution.
Proteins can be seen as existing within an organized space where each protein is neighbored by every other that is one mutational event away (54). The structure of evolution appears as a network within this space, connecting all proteins by the paths that evolution can take between them. The paths that evolution can follow are the ones by which each protein transforms into the next without the collective loss of function of the system it is a part of. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - Proteins exist within an organized space.
It is in this space that a language model sees proteins. It sees the data of proteins as filling this space, densely in some regions, and sparsely in others, revealing the parts that are accessible to evolution. Since the next token is generated by evolution, it follows that to solve the training task of predicting the next token, a language model must predict how evolution moves through the space of possible proteins. To do so it will need to learn what determines whether a path is feasible for evolution. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - A language model perceimates proteins in a defined space.
Simulations are computational representations of reality. In that sense a language model which can predict possible outcomes of evolution can be said to be a simulator of it. ESM3 is an emergent simulator that has been learned from solving a token prediction task on data generated by evolution. It has been theorized that neural networks discover the underlying structure of the data they are trained to predict $(55,56)$. In this way, solving the token prediction task would require the model to learn the deep structure that determines which steps evolution can take, i.e. the fundamental biology of proteins. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - Simulations are computational representations of reality.
In ESM3's generation of a new fluorescent protein, it is the first chain of thought to B8 that is the most intriguing. At 96 mutations to B8's closest neighbor there are $\binom{229}{96} \times 19^{96}$ possible proteins, an astronomical number out of which only a vanishingly small fraction can have function, since fluorescence falls off sharply even after just a few random mutations. The existence of $\mathrm{C} 10$ and other bright designs in the neighborhood of B8 confirms that in the first chain of thought to B8, ESM3 found a new part of the space of proteins that, although unexplored by nature, is dense with fluorescent proteins. Please extract each individual fact or idea from this paragraph and display in markdown list format
AI: - In ESM3's generation of a new fluorescent protein, the initial thought process leading to B8 is particularly notable.
Please examine and analyze each of the points made in this discussion section and critique them as if you were a indepentent scientist reviewing an article for Nature
AI: As an independent scientist tasked with reviewing this theoretical exposition for Nature, my evaluation is as follows:
In conclusion, while the proposed work is pioneering and the implications for both evolutionary biology and protein engineering are profound, a deeper delve into methodological specifics, alongside a more nuanced discussion of the model's limitations and the biological realities it either reflects or overlooks, would enhance the discourse. Such critique is intended not as a disparagement but as a guide for refining and strengthening the argument for a scholarly audience.
Please review as though a reviewer for Nature
AI: Review for Manuscript Submission to Nature:
Title: [To be provided by the authors]
Authors: [To be provided by the authors]
Manuscript ID: [To be provided by the authors]
General Assessment: The manuscript presents a fascinating study on the use of protein language models, specifically ESM3, in predicting and realizing the structure of proteins that lie beyond the space typically explored by natural evolution. By leveraging the immense combinatorial space of protein sequences, the authors demonstrate that these models not only predict but can also generate functional proteins with properties evolution has not explored within observable timeframes. The implication of this capability extends from fundamental understanding of evolutionary biology to practical applications in biotechnology and medicine. However, there are several areas in which the manuscript could be strengthened to meet the publication standards of Nature.
Major Comments:
Novelty and Significance: The manuscript intriguingly posits that language models like ESM3 can operate as "emergent simulators" of evolution, a concept both novel and with far-reaching implications. However, the manuscript could better articulate how this work advances beyond the current state-of-the-art in protein design and evolutionary biology. Specifically, clarifying how ESM3's approach and outcomes compare to existing evolutionary simulation methods would greatly enhance the manuscript's impact.
Methodological Rigor: The text provides a high-level overview of how EMA3 predicts and generates novel protein structures but stops short of offering detailed methodological insight. For a broader scientific audience and in accordance with the standards of Nature, a more comprehensive breakdown of the model architecture, training data, and computational processes would be beneficial. Additionally, the rationale behind choosing specific proteins (e.g., fluorescent proteins) for study should be better integrated into the manuscript.
Experimental Validation: While the computational findings are compelling, the manuscript would be significantly strengthened by the inclusion of experimental validation of the predicted proteins' functionality. The generation of fluorescent proteins is particularly striking; however, details on the biochemical verification of these proteins’ fluorescence and functionality are sparse. Describing the experimental methodologies and outcomes in greater detail would substantially bolster the claims made.
Theoretical Implications and Discussion: The authors position their work within a broad theoretical framework, suggesting that neural networks can uncover the underlying structure of evolutionary biology. Further exposition on how ESM3’s findings could influence our understanding of evolutionary constraints, protein fitness landscapes, and evolutionary theory more broadly would enrich the discussion. The potential limitations of extrapolating from ESM3’s predictions to broader evolutionary phenomena should also be addressed.
Accessibility and Reproducibility: To adhere to Nature's standards on data availability and computational reproducibility, the manuscript should include more explicit details about dataset accessibility, model code, and the parameters used for training ESM3. This information is crucial for enabling replication studies and fostering further research in this vital area.
Minor Comments:
Literature Review: The manuscript would benefit from a more detailed review of related work, placing its contributions in context with existing literature on protein design, evolutionary simulations, and the application of language models in biological research.
Statistical Analysis: Where applicable, a more detailed description of the statistical analyses used to evaluate ESM3’s predictions and the resultant protein functionalities would be informative.
Clarification of Terms: Certain terms and concepts, such as “token prediction task,” “deep structure,” and “evolutionary space,” are used throughout the manuscript but could be better defined for readers not familiar with computational biology or machine learning.
Conclusion: The manuscript under review represents an ambitious and potentially groundbreaking approach to understanding and leveraging the vast spaces of protein functionality unexplored by natural evolution. While the claims are of significant interest, the manuscript would benefit from further detail, clarification, and validation to meet the publication standards of Nature. With these revisions, the work has the potential to make a substantial contribution to the fields of evolutionary biology, computational biology, and protein engineering.
We thank Eric Schreiter, Karel Svoboda, and Srinivas Turaga for feedback on the properties of esmGFP. We thank Marko Iskander, Vishvajit Kher, and the Andromeda cluster team for support on compute infrastructure. We thank April Pawluk for assistance with manuscript preparation. We also thank the experts who provided feedback on our approach to responsible development, and the experts who participated in the review of the risks and benefits of releasing ESM3-open.
Data: H.A., Z.L., R.R., A.R., T.S., N.T., R.V.
Pre-training: H.A., S.C., J.D., T.H., Z.L., D.O., R.R., A.R., T.S., I.S., R.V., M.W.
Post-training: H.A., S.C., A.D., J.G., T.H., D.O., R.R., A.R., M.W.
Evaluation and Analysis: R.B., J.D., A.D., T.H., Y.K., C.K., Z.L., R.S.M., A.R., N.J.S.
Open Model \& Responsible Development: J.G., I.S.,
N.J.S., T.S., R.S.M., Z.L., R.R., A.R., N.T.
API \& Deployment: J.G., C.M., R.S.M., Z.L., T.S.
GFP Computational: S.C., T.H., N.J.S., A.R., R.V.
GFP Experimental Validation: L.J.B., P.D.H., Y.K., N.J.S., N.T., V.Q.T.
Authors H.A., R.B., S.C., J.D., A.D., J.G., T.H., C.K., Z.L., R.S.M., C.M., D.O., R.R., A.R., N.J.S., T.S., I.S., N.T., R.V., M.W. are employees of EvolutionaryScale, PBC. P.D.H. is a cofounder of Stylus Medicine, Circle Labs, and Spotlight Therapeutics, serves on the board of directors at Stylus Medicine, is a board observer at EvolutionaryScale, Circle Labs, and Spotlight Therapeutics, a scientific advisory board member at Arbor Biosciences and Veda Bio, and an advisor to NFDG, Varda Space, and Vial Health. Patents have been filed related to aspects of this work.
Weights and code for ESM3-open are provided for academic research use. The ESM3-open model was reviewed by a committee of technical experts who found that the benefits of releasing the model greatly outweighed any potential risks. ESM3 models will be available via API with a free access tier for academic research. The sequence of esmGFP (along with the other GFPs generated for this work) is committed to the public domain. Plasmids for esmGFP-C10 and esmGFP-B8 will be made available.
[1] UniProt Consortium. Uniprot: a hub for protein information. Nucleic acids research, 43(D1):D204$\mathrm{D} 212,2015$. Please convert into a bibtex citation.
AI:
@article{UniProtConsortium2015,
title={UniProt: a hub for protein information},
author={UniProt Consortium},
journal={Nucleic Acids Research},
volume={43},
number={D1},
pages={D204--D212},
year={2015},
publisher={Oxford University Press}
}
[2] Igor V Grigoriev, Henrik Nordberg, Igor Shabalov, Andrea Aerts, Mike Cantor, David Goodstein, Alan Kuo, Simon Minovitsky, Roman Nikitin, Robin A Ohm, et al. The genome portal of the department of energy joint genome institute. Nucleic acids research, 40(D1):D26-D32, 2012. Please convert into a bibtex citation.
AI: Here is the requested citation in BibTeX format:
@article{Grigoriev2012TheGP,
title={The genome portal of the Department of Energy Joint Genome Institute},
author={Igor V. Grigoriev and Henrik Nordberg and Igor Shabalov and Andrea Aerts and Mike Cantor and David Goodstein and Alan Kuo and Simon Minovitsky and Roman Nikitin and Robin A. Ohm and et al.},
journal={Nucleic Acids Research},
volume={40},
number={D1},
pages={D26-D32},
year={2012},
publisher={Oxford Univ Press}
}
Remember that the et al. in the author field is not standard BibTeX format and typically, you would list all authors or use a predefined threshold after which you abbreviate the list with and others to represent additional authors, depending on the citation guidelines of the specific bibliography you are adhering to. However, for example purposes or informal use, this could be permissible as shown, depending on your needs.
[3] Alex L Mitchell, Alexandre Almeida, Martin Beracochea, Miguel Boland, Josephine Burgin, Guy Cochrane, Michael R Crusoe, Varsha Kale, Simon C Potter, Lorna J Richardson, Ekaterina Sakharova, Maxim Scheremetjew, Anton Korobeynikov, Alex Shlemov, Olga Kunyavskaya, Alla Lapidus, and Robert D Finn. MGnify: the microbiome analysis resource in 2020. Nucleic Acids Research, 48(D1): D570-D578, January 2020. ISSN 0305-1048. doi: 10.1093/nar/gkz1035. URL https://doi.org/ 10.1093/nar/gkz1035. Please convert into a bibtex citation.
AI:
@article{Mitchell2020,
title={MGnify: the microbiome analysis resource in 2020},
author={Mitchell, Alex L and Almeida, Alexandre and Beracochea, Martin and Boland, Miguel and Burgin, Josephine and Cochrane, Guy and Crusoe, Michael R and Kale, Varsha and Potter, Simon C and Richardson, Lorna J and Sakharova, Ekaterina and Scheremetjew, Maxim and Korobeynikov, Anton and Shlemov, Alex and Kunyavskaya, Olga and Lapidus, Alla and Finn, Robert D},
journal={Nucleic Acids Research},
volume={48},
number={D1},
pages={D570-D578},
year={2020},
publisher={Oxford University Press},
doi={10.1093/nar/gkz1035},
issn={0305-1048},
url={https://doi.org/10.1093/nar/gkz1035}
}
[4] Mihaly Varadi, Damian Bertoni, Paulyna Magana, Urmila Paramval, Ivanna Pidruchna, Malarvizhi Radhakrishnan, Maxim Tsenkov, Sreenath Nair, Milot Mirdita, Jingi Yeo, Oleg Kovalevskiy, Kathryn Tunyasuvunakool, Agata Laydon, Augustin Žídek, Hamish Tomlinson, Dhavanthi Hariharan, Josh Abrahamson, Tim Green, John Jumper, Ewan Birney, Martin Steinegger, Demis Hassabis, and Sameer Velankar. AlphaFold Protein Structure Database in 2024: providing structure coverage for over 214 million protein sequences. Nucleic Acids Research, 52(D1): D368-D375, January 2024. ISSN 1362-4962. doi: 10.1093/nar/gkad1011. Please convert into a bibtex citation.
AI:
@article{Varadi2024,
author = {Mihaly Varadi and Damian Bertoni and Paulyna Magana and Urmila Paramval and Ivanna Pidruchna and Malarvizhi Radhakrishnan and Maxim Tsenkov and Sreenath Nair and Milot Mirdita and Jingi Yeo and Oleg Kovalevskiy and Kathryn Tunyasuvunakool and Agata Laydon and Augustin Žídek and Hamish Tomlinson and Dhavanthi Hariharan and Josh Abrahamson and Tim Green and John Jumper and Ewan Birney and Martin Steinegger and Demis Hassabis and Sameer Velankar},
title = {AlphaFold Protein Structure Database in 2024: providing structure coverage for over 214 million protein sequences},
journal = {Nucleic Acids Research},
volume = {52},
number = {D1},
pages = {D368-D375},
year = {2024},
doi = {10.1093/nar/gkad1011},
issn = {1362-4962}
}
[5] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637): $1123-1130,2023$. Please convert into a bibtex citation.
AI:
@article{Lin2023,
title={Evolutionary-scale prediction of atomic-level protein structure with a language model},
author={Lin, Zeming and Akin, Halil and Rao, Roshan and Hie, Brian and Zhu, Zhongkai and Lu, Wenting and Smetanin, Nikita and Verkuil, Robert and Kabeli, Ori and Shmueli, Yaniv and others},
journal={Science},
volume={379},
number={6637},
pages={1123--1130},
year={2023}
}
[6] Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M Church. Unified rational protein engineering with sequence-based deep representation learning. Nature Methods, 16 (12):1-8, 2019. Please convert into a bibtex citation.
AI:
@article{Alley2019UnifiedRP,
title={Unified rational protein engineering with sequence-based deep representation learning},
author={Ethan C. Alley and Grigory Khimulya and Surojit Biswas and Mohammed AlQuraishi and George M. Church},
journal={Nature Methods},
volume={16},
number={12},
pages={1-8},
year={2019}
}
[7] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15):e2016239118, April 2021. ISSN 0027-8424, 1091-6490. doi: 10.1073/pnas. 2016239118. URL https://www.pnas.org/ content/118/15/e2016239118. Publisher: National Academy of Sciences Section: Biological Sciences. Please convert into a bibtex citation.
AI:
@article{Rives2021,
title={Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences},
author={Rives, Alexander and Meier, Joshua and Sercu, Tom and Goyal, Siddharth and Lin, Zeming and Liu, Jason and Guo, Demi and Ott, Myle and Zitnick, C Lawrence and Ma, Jerry and others},
journal={Proceedings of the National Academy of Sciences},
volume={118},
number={15},
pages={e2016239118},
year={2021},
month={April},
publisher={National Academy of Sciences},
ISSN={0027-8424, 1091-6490},
doi={10.1073/pnas.2016239118},
url={https://www.pnas.org/content/118/15/e2016239118},
section={Biological Sciences}
}
[8] Ali Madani, Ben Krause, Eric R. Greene, Subu Subramanian, Benjamin P. Mohr, James M. Holton, Jose Luis Olmos, Caiming Xiong, Zachary Z. Sun, Richard Socher, James S. Fraser, and Nikhil Naik. Large language models generate functional protein sequences across diverse families. Nature Biotechnology, 41(8):1099-1106, August 2023. ISSN 1546-1696. doi: 10.1038/s41587-022-01618-2. URL https://www.nature.com/articles/ s41587-022-01618-2. Publisher: Nature Publishing Group. Please convert into a bibtex citation.
AI:
@article{Madani2023LargeLM,
title={Large language models generate functional protein sequences across diverse families},
author={Ali Madani and Ben Krause and Eric R. Greene and Subu Subramanian and Benjamin P. Mohr and James M. Holton and Jose Luis Olmos and Caiming Xiong and Zachary Z. Sun and Richard Socher and James S. Fraser and Nikhil Naik},
journal={Nature Biotechnology},
volume={41},
number={8},
pages={1099--1106},
year={2023},
month={Aug},
publisher={Nature Publishing Group},
doi={10.1038/s41587-022-01618-2},
issn={1546-1696},
url={https://www.nature.com/articles/s41587-022-01618-2}
}
[9] Noelia Ferruz, Steffen Schmidt, and Birte Höcker. ProtGPT2 is a deep unsupervised language model for protein design. Nat. Commun., 13(1):4348, July 2022. Please convert into a bibtex citation.
AI:
@article{Ferruz2022ProtGPT2,
title={ProtGPT2 is a deep unsupervised language model for protein design},
author={Ferruz, Noelia and Schmidt, Steffen and Höcker, Birte},
journal={Nat. Commun.},
volume={13},
number={1},
pages={4348},
year={2022},
month={Jul},
publisher={Nature Publishing Group}
}
[10] Robert Verkuil, Ori Kabeli, Yilun Du, Basile IM Wicky, Lukas F Milles, Justas Dauparas, David Baker, Sergey Ovchinnikov, Tom Sercu, and Alexander Rives. Language models generalize beyond natural proteins. bioRxiv, pages 2022-12, 2022. Please convert into a bibtex citation.
AI:
@article{verkuil2022language,
title={Language models generalize beyond natural proteins},
author={Verkuil, Robert and Kabeli, Ori and Du, Yilun and Wicky, Basile IM and Milles, Lukas F and Dauparas, Justas and Baker, David and Ovchinnikov, Sergey and Sercu, Tom and Rives, Alexander},
journal={bioRxiv},
pages={2022-12},
year={2022}
}
[11] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Debsindhu Bhowmik, and Burkhard Rost. ProtTrans: Towards Cracking the Language of Lifes Code Through Self-Supervised Deep Learning and High Performance Computing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(8):1-1, July 2021. doi: 10.1109/TPAMI. 2021.3095381. URL https://www.osti.gov/ pages/biblio/1817585. Institution: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Please convert into a bibtex citation.
AI:
@article{Elnaggar2021ProtTrans,
title={ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing},
author={Ahmed Elnaggar and Michael Heinzinger and Christian Dallago and Ghalia Rihawi and Yu Wang and Llion Jones and Tom Gibbs and Tamas Feher and Christoph Angerer and Debsindhu Bhowmik and Burkhard Rost},
journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
volume={14},
number={8},
pages={1-1},
year={2021},
month={July},
doi={10.1109/TPAMI.2021.3095381},
url={https://www.osti.gov/pages/biblio/1817585},
institution={Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)}
}
[12] Daniel Hesslow, Niccoló Zanichelli, Pascal Notin, Iacopo Poli, and Debora Marks. RITA: a Study on Scaling Up Generative Protein Sequence Models, July 2022. URL http: / / arxiv.org/abs / 2205.0578 9. arXiv:2205.05789 [cs, q-bio]. Please convert into a bibtex citation.
AI:
@article{Hesslow2022RITA,
title={RITA: a Study on Scaling Up Generative Protein Sequence Models},
author={Hesslow, Daniel and Zanichelli, Niccoló and Notin, Pascal and Poli, Iacopo and Marks, Debora},
journal={arXiv preprint arXiv:2205.05789},
year={2022},
month={jul},
url={http://arxiv.org/abs/2205.05789},
archivePrefix={arXiv},
eprint={2205.05789},
primaryClass={cs, q-bio}
}
[13]
[14] Sarah Alamdari, Nitya Thakkar, Rianne van den Berg, Alex Xijie Lu, Nicolo Fusi, Ava Pardis Amini, and Kevin K Yang. Protein generation with evolutionary diffusion: sequence is all you need. bioRxiv, pages 2023-09, 2023. Please convert into a bibtex citation.
AI:
@article{Alamdari2023Protein,
title = {Protein generation with evolutionary diffusion: sequence is all you need},
author = {Alamdari, Sarah and Thakkar, Nitya and van den Berg, Rianne and Lu, Alex Xijie and Fusi, Nicolo and Amini, Ava Pardis and Yang, Kevin K},
journal = {bioRxiv},
pages = {2023-09},
year = {2023}
}
[15] Michael Heinzinger, Ahmed Elnaggar, Yu Wang, Christian Dallago, Dmitrii Nechaev, Florian Matthes, and Burkhard Rost. Modeling aspects of the language of life through transfer-learning protein sequences. BMC bioinformatics, 20(1):723, 2019. Please convert into a bibtex citation.
AI:
@article{Heinzinger2019ModelingAO,
title={Modeling aspects of the language of life through transfer-learning protein sequences},
author={Michael Heinzinger and Ahmed Elnaggar and Yu Wang and Christian Dallago and Dmitrii Nechaev and Florian Matthes and Burkhard Rost},
journal={BMC Bioinformatics},
volume={20},
number={1},
pages={723},
year={2019},
publisher={BioMed Central}
}
[16] Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. Advances in Neural Information Processing Systems, 34, July 2021. doi: 10.1101/2021.07.09.450648. URL http://biorxiv.org/lookup/doi/10. $1101 / 2021.07 .09 .450648$. Please convert into a bibtex citation.
AI: Certainly! Here's how you can format this reference into a BibTeX entry:
@article{Meier2021LanguageModels,
author = {Joshua Meier and Roshan Rao and Robert Verkuil and Jason Liu and Tom Sercu and Alex Rives},
title = {Language models enable zero-shot prediction of the effects of mutations on protein function},
journal = {Advances in Neural Information Processing Systems},
volume = {34},
year = {2021},
month = {July},
doi = {10.1101/2021.07.09.450648},
url = {http://biorxiv.org/lookup/doi/10.1101/2021.07.09.450648}
}
When using this BibTeX citation, make sure to adjust it according to the specific requirements (like entry identifier) and consistency (e.g., use of brackets) of your document.
[17] Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, and Alexander Rives. Transformer protein language models are unsupervised structure learners. In International Conference on Learning Representations, page 2020.12.15.422761. Cold Spring Harbor Laboratory, December 2021. doi: $10.1101 / 2020.12 .15 .422761$. Please convert into a bibtex citation.
AI: To convert the provided reference into a BibTeX citation, you would format it roughly as follows. Please note, for accuracy, double-check specifics such as conference naming and DOI formatting according to the requirements of your referencing system or target journal.
@inproceedings{Rao2021Transformer,
title={Transformer protein language models are unsupervised structure learners},
author={Rao, Roshan and Meier, Joshua and Sercu, Tom and Ovchinnikov, Sergey and Rives, Alexander},
booktitle={International Conference on Learning Representations},
pages={422761},
year={2021},
month={12},
organization={Cold Spring Harbor Laboratory},
doi={10.1101/2020.12.15.422761},
note={Accessed: yyyy-mm-dd}
}
A few adjustments might be required depending on specific citation style guidelines or the version/format of BibTeX you're using. Here, note={Accessed: yyyy-mm-dd} is optional and can be included to specify the date you accessed the document if it's an online source. Make sure to replace yyyy-mm-dd with the actual date of access.
Remember to adjust the details (like page numbers, conference name format, etc.) as per the exact specifications needed for your purposes.
[18] Bo Chen, Xingyi Cheng, Li-ao Gengyang, Shen Li, Xin Zeng, Boyan Wang, Gong Jing, Chiming Liu, Aohan Zeng, Yuxiao Dong, et al. xtrimopglm: Unified $100 b$-scale pre-trained transformer for deciphering the language of protein. bioRxiv, pages 2023-07, 2023. Please convert into a bibtex citation.
@article{chen2023xtrimopglm, title={xtrimopglm: Unified $100 b$-scale pre-trained transformer for deciphering the language of protein}, author={Chen, Bo and Cheng, Xingyi and Gengyang, Li-ao and Li, Shen and Zeng, Xin and Wang, Boyan and Jing, Gong and Liu, Chiming and Zeng, Aohan and Dong, Yuxiao and others}, journal={bioRxiv}, pages={2023-07}, year={2023} }
[19] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models, January 2020. URL http://arxiv.org/abs/2001. 08361. arXiv:2001.08361 [cs, stat].
Please convert into a bibtex citation.
[AI]:
bibtex @article{Kaplan2020ScalingLF, title={Scaling Laws for Neural Language Models}, author={Jared Kaplan and Sam McCandlish and Tom Henighan and Tom B. Brown and Benjamin Chess and Rewon Child and Scott Gray and Alec Radford and Jeffrey Wu and Dario Amodei}, journal={arXiv preprint arXiv:2001.08361}, year={2020}, month={Jan}, url={http://arxiv.org/abs/2001.08361}, archivePrefix={arXiv}, eprint={2001.08361}, primaryClass={cs, stat} }
[20] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are FewShot Learners. CoRR, abs/2005.14165:1877-1901, 2020. URL https://arxiv.org/abs/2005. 14165. _eprint: 2005.14165.
Please convert into a bibtex citation.
[AI]: Certainly! Here's the BibTeX citation for the paper titled "Language Models are Few-Shot Learners" by Tom B. Brown et al.:
bibtex @article{brown2020language, title={Language Models are FewShot Learners}, author={Brown, Tom B. and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel M. and Wu, Jeffrey and Winter, Clemens and Hesse, Christopher and Chen, Mark and Sigler, Eric and Litwin, Mateusz and Gray, Scott and Chess, Benjamin and Clark, Jack and Berner, Christopher and McCandlish, Sam and Radford, Alec and Sutskever, Ilya and Amodei, Dario}, journal={CoRR}, volume={abs/2005.14165}, pages={1877--1901}, year={2020}, url={https://arxiv.org/abs/2005.14165}, eprint={2005.14165} }
[21] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training ComputeOptimal Large Language Models. March 2022. doi: 10.48550/arXiv.2203.15556. URL https: //arxiv.org/abs/2203.15556v1.
Please convert into a bibtex citation.
[AI]:
bibtex @article{Hoffmann2022TrainingCO, title={Training Compute-Optimal Large Language Models}, author={Jordan Hoffmann and Sebastian Borgeaud and Arthur Mensch and Elena Buchatskaya and Trevor Cai and Eliza Rutherford and Diego de Las Casas and Lisa Anne Hendricks and Johannes Welbl and Aidan Clark and Tom Hennigan and Eric Noland and Katie Millican and George van den Driessche and Bogdan Damoc and Aurelia Guy and Simon Osindero and Karen Simonyan and Erich Elsen and Jack W. Rae and Oriol Vinyals and Laurent Sifre}, journal={arXiv}, year={2022}, volume={abs/2203.15556}, url={https://arxiv.org/abs/2203.15556v1}, doi={10.48550/arXiv.2203.15556} }
[22] Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J. Ballard, Joshua Bambrick, Sebastian W. Bodenstein, David A. Evans, Chia-Chun Hung, Michael O'Neill, David Reiman, Kathryn Tunyasuvunakool, Zachary Wu, Akvilè Žemgulytė, Eirini Arvaniti, Charles Beattie, Ottavia Bertolli, Alex Bridgland, Alexey Cherepanov, Miles Congreve, Alexander I. Cowen-Rivers, Andrew Cowie, Michael Figurnov, Fabian B. Fuchs, Hannah Gladman, Rishub Jain, Yousuf A. Khan, Caroline M. R. Low, Kuba
Please convert into a bibtex citation.
[AI]: Certainly! However, it appears you have listed names without specifying the corresponding work (such as a paper title, journal name, publication year, etc.). To generate a proper BibTeX citation, I need a bit more detail. Given the nature of your request, I'll create a generic BibTeX template for an article with placeholders for you to fill in. Once you have the missing details, you can update the template accordingly.
bibtex @article{AuthorYear, author = {Josh Abramson and Jonas Adler and Jack Dunger and Richard Evans and Tim Green and Alexander Pritzel and Olaf Ronneberger and Lindsay Willmore and Andrew J. Ballard and Joshua Bambrick and Sebastian W. Bodenstein and David A. Evans and Chia-Chun Hung and Michael O'Neill and David Reiman and Kathryn Tunyasuvunakool and Zachary Wu and Akvilè Žemgulytė and Eirini Arvaniti and Charles Beattie and Ottavia Bertolli and Alex Bridgland and Alexey Cherepanov and Miles Congreve and Alexander I. Cowen-Rivers and Andrew Cowie and Michael Figurnov and Fabian B. Fuchs and Hannah Gladman and Rishub Jain and Yousuf A. Khan and Caroline M. R. Low and Kuba}, title = {The Title of the Work}, journal = {Journal Name}, year = {Year of Publication}, volume = {Volume Number}, number = {Issue Number}, pages = {Page Numbers}, month = {Month of Publication}, doi = {DOI}, url = {URL} }
Please replace the placeholders (`The Title of the Work`, `Journal Name`, `Year of Publication`, `Volume Number`, `Issue Number`, `Page Numbers`, `Month of Publication`, `DOI`, `URL`) with the specific details of the work you're referencing. If you provide the additional details, I can help tailor the citation more accurately.
Perlin, Anna Potapenko, Pascal Savy, Sukhdeep Singh, Adrian Stecula, Ashok Thillaisundaram, Catherine Tong, Sergei Yakneen, Ellen D. Zhong, Michal Zielinski, Augustin Žídek, Victor Bapst, Pushmeet Kohli, Max Jaderberg, Demis Hassabis, and John M. Jumper. Accurate structure prediction of biomolecular interactions with AlphaFold 3. Nature, 630(8016):493-500, June 2024. ISSN 14764687. doi: 10.1038/s41586-024-07487-w. URL
Please convert into a bibtex citation.
[AI]: Certainly! Below is the requested citation in BibTeX format. Please note that for future use or within real publications, you might want to check the details for accuracy and completeness.
bibtex @article{Perlin2024Accurate, title={Accurate structure prediction of biomolecular interactions with AlphaFold 3}, author={Perlin, Anna and Potapenko, Anna and Savy, Pascal and Singh, Sukhdeep and Stecula, Adrian and Thillaisundaram, Ashok and Tong, Catherine and Yakneen, Sergei and Zhong, Ellen D. and Zielinski, Michal and Žídek, Augustin and Bapst, Victor and Kohli, Pushmeet and Jaderberg, Max and Hassabis, Demis and Jumper, John M.}, journal={Nature}, volume={630}, number={8016}, pages={493--500}, year={2024}, month={Jun}, issn={14764687}, doi={10.1038/s41586-024-07487-w}, url={http://dx.doi.org/10.1038/s41586-024-07487-w} }
This BibTeX entry includes all the elements provided: authors, title, journal, volume, number, pages, year, month, ISSN, DOI, and the URL. Ensure to integrate this citation into your LaTeX documents appropriately, and if this is for real academic work, always verify the entry for accuracy against the source.
https://www.nature.com/articles/ s41586-024-07487-w. Publisher: Nature Publishing Group.
[23] Joseph L. Watson, David Juergens, Nathaniel R. Bennett, Brian L. Trippe, Jason Yim, Helen E. Eisenach, Woody Ahern, Andrew J. Borst, Robert J. Ragotte, Lukas F. Milles, Basile I. M. Wicky, Nikita Hanikel, Samuel J. Pellock, Alexis Courbet, William Sheffler, Jue Wang, Preetham Venkatesh, Isaac Sappington, Susana Vázquez Torres, Anna Lauko, Valentin De Bortoli, Emile Mathieu, Sergey Ovchinnikov, Regina Barzilay, Tommi S. Jaakkola, Frank DiMaio, Minkyung Baek, and David Baker. De novo design of protein structure and function with RFdiffusion. Nature, 620(7976):1089-1100, August 2023. ISSN 1476-4687. doi: 10.1038/s41586-023-06415-8. URL https://www.nature.com/articles/ s41586-023-06415-8. Publisher: Nature Publishing Group.
Please convert into a bibtex citation.
[AI]: Certainly! Below is the mentioned citation in BibTeX format:
bibtex @article{Watson2023, author = {Watson, Joseph L. and Juergens, David and Bennett, Nathaniel R. and Trippe, Brian L. and Yim, Jason and Eisenach, Helen E. and Ahern, Woody and Borst, Andrew J. and Ragotte, Robert J. and Milles, Lukas F. and Wicky, Basile I. M. and Hanikel, Nikita and Pellock, Samuel J. and Courbet, Alexis and Sheffler, William and Wang, Jue and Venkatesh, Preetham and Sappington, Isaac and V{\'a}zquez Torres, Susana and Lauko, Anna and De Bortoli, Valentin and Mathieu, Emile and Ovchinnikov, Sergey and Barzilay, Regina and Jaakkola, Tommi S. and DiMaio, Frank and Baek, Minkyung and Baker, David}, title = {De novo design of protein structure and function with RFdiffusion}, journal = {Nature}, volume = {620}, number = {7976}, pages = {1089--1100}, year = {2023}, month = {Aug}, doi = {10.1038/s41586-023-06415-8}, url = {https://www.nature.com/articles/s41586-023-06415-8}, publisher = {Nature Publishing Group}, issn = {1476-4687} }
[24] John B. Ingraham, Max Baranov, Zak Costello, Karl W. Barber, Wujie Wang, Ahmed Ismail, Vincent Frappier, Dana M. Lord, Christopher Ng-Thow-Hing, Erik R. Van Vlack, Shan Tie, Vincent Xue, Sarah C. Cowles, Alan Leung, João V. Rodrigues, Claudio L. Morales-Perez, Alex M. Ayoub, Robin Green, Katherine Puentes, Frank Oplinger, Nishant V. Panwar, Fritz Obermeyer, Adam R. Root, Andrew L. Beam, Frank J. Poelwijk, and Gevorg Grigoryan. Illuminating protein space with a programmable generative model. Nature, 623(7989):1070-1078, November 2023. ISSN 1476-4687. doi: 10.1038/s41586-023-06728-8. URL https://www.nature.com/articles/ s41586-023-06728-8. Publisher: Nature Publishing Group.
Please convert into a bibtex citation.
[AI]:
bibtex @article{Ingraham2023, author = {Ingraham, John B. and Baranov, Max and Costello, Zak and Barber, Karl W. and Wang, Wujie and Ismail, Ahmed and Frappier, Vincent and Lord, Dana M. and Ng-Thow-Hing, Christopher and Van Vlack, Erik R. and Tie, Shan and Xue, Vincent and Cowles, Sarah C. and Leung, Alan and Rodrigues, Jo{~a}o V. and Morales-Perez, Claudio L. and Ayoub, Alex M. and Green, Robin and Puentes, Katherine and Oplinger, Frank and Panwar, Nishant V. and Obermeyer, Fritz and Root, Adam R. and Beam, Andrew L. and Poelwijk, Frank J. and Grigoryan, Gevorg}, title = {Illuminating protein space with a programmable generative model}, journal = {Nature}, year = {2023}, volume = {623}, number = {7989}, pages = {1070-1078}, month = {Nov}, issn = {1476-4687}, doi = {10.1038/s41586-023-06728-8}, url = {https://www.nature.com/articles/s41586-023-06728-8}, publisher = {Nature Publishing Group} }
[25] Yeqing Lin, Minji Lee, Zhao Zhang, and Mohammed AlQuraishi. Out of many, one: Designing and scaffolding proteins at the scale of the structural universe with genie 2, may 2024. URL https: //arxiv.org/abs/2405.15489.
Please convert into a bibtex citation.
[AI]: Here is the BibTeX citation for the document you provided:
bibtex @misc{lin2024out, title={Out of Many, One: Designing and Scaffolding Proteins at the Scale of the Structural Universe with Genie 2}, author={Lin, Yeqing and Lee, Minji and Zhang, Zhao and AlQuraishi, Mohammed}, year={2024}, month={may}, url={https://arxiv.org/abs/2405.15489} }
Please note, in the context of publication year and format specifics (e.g., it being a preprint on arXiv), I've categorized this entry under `@misc`, following the traditional BibTeX documentation. Adjust the citation type (`@article`, `@inproceedings`, etc.) as needed based on the context of where and how the document is being referenced or published.
[26] Osamu Shimomura, Frank H. Johnson, and Yo Saiga. Extraction, purification and properties of aequorin, a bioluminescent protein from the luminous hydromedusan, aequorea. Journal of Cellular and Comparative Physiology, 59(3):223-239, 1962. doi: https://doi.org/10.1002/jcp.1030590302. URL https://onlinelibrary.wiley.com/ doi/abs/10.1002/jcp. 1030590302.
Please convert into a bibtex citation.
[AI]:
bibtex @article{Shimomura1962, title={Extraction, purification and properties of aequorin, a bioluminescent protein from the luminous hydromedusan, aequorea}, author={Shimomura, Osamu and Johnson, Frank H. and Saiga, Yo}, journal={Journal of Cellular and Comparative Physiology}, volume={59}, number={3}, pages={223-239}, year={1962}, doi={10.1002/jcp.1030590302}, url={https://onlinelibrary.wiley.com/doi/abs/10.1002/jcp.1030590302} }
[27] R. Y. Tsien. The green fluorescent protein. Annual Review of Biochemistry, 67:509-544, 1998. ISSN 0066-4154. doi: 10.1146/annurev.biochem.67.1.509.
Please convert into a bibtex citation.
[AI]:
bibtex @article{Tsien1998, title={The green fluorescent protein}, author={Tsien, Roger Y}, journal={Annual Review of Biochemistry}, volume={67}, pages={509--544}, year={1998}, publisher={Annual Reviews}, issn={0066-4154}, doi={10.1146/annurev.biochem.67.1.509} }
[28] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North $\{A\}$ merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL http: //arxiv.org/abs/1810.04805.
Please convert into a bibtex citation.
[AI]: Certainly! Here's a BibTeX citation for the reference you've provided:
bibtex @inproceedings{devlin2019bert, title={BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding}, author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina}, booktitle={Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies}, volume={1}, pages={4171--4186}, year={2019}, organization={Association for Computational Linguistics}, address={Minneapolis, Minnesota}, month={June}, doi={10.18653/v1/N19-1423}, url={http://arxiv.org/abs/1810.04805} }
Please ensure to adjust formatting according to the specific requirements of the citation style or publication venue you're preparing this entry for.
[29] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
Please convert into a bibtex citation.
[AI]:
bibtex @inproceedings{chang2022maskgit, title={Maskgit: Masked generative image transformer}, author={Chang, Huiwen and Zhang, Han and Jiang, Lu and Liu, Ce and Freeman, William T}, booktitle={The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, month={June}, year={2022} }
[30] Benigno Uria, Iain Murray, and Hugo Larochelle. A deep and tractable density estimator. In Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32, ICML'14, page I-467-I-475. JMLR.org, 2014.
Please convert into a bibtex citation.
[AI]: Here is the citation in BibTeX format:
bibtex @inproceedings{Uria2014DeepTractable, title={A deep and tractable density estimator}, author={Uria, Benigno and Murray, Iain and Larochelle, Hugo}, booktitle={Proceedings of the 31st International Conference on Machine Learning - Volume 32}, pages={I-467--I-475}, year={2014}, organization={JMLR.org}, series = {ICML'14}, }
[31] Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces, 2023 .
Please convert into a bibtex citation.
[AI]:
bibtex @article{Austin2023StructuredDD, title={Structured Denoising Diffusion Models in Discrete State-Spaces}, author={Jacob Austin and Daniel D. Johnson and Jonathan Ho and Daniel Tarlow and Rianne van den Berg}, journal={arXiv}, year={2023}, }
Note: Since the exact journal or conference isn't specified beyond being an `arXiv` preprint in the original information provided, it is classified under `@article` with `journal={arXiv}`. This can be updated to more specific details once the paper is published in a peer-reviewed venue.
[32] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. Advances in Neural Information Processing Systems, 2017.
Please convert into a bibtex citation.
[AI]:
bibtex @inproceedings{van2017neural, title={Neural discrete representation learning}, author={van den Oord, Aaron and Vinyals, Oriol and Kavukcuoglu, Koray}, booktitle={Advances in Neural Information Processing Systems}, year={2017} }
[33] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and Memory-Efficient Exact Attention with IOAwareness, June 2022. URL http://arxiv. org/abs/2205 . 14135. arXiv:2205.14135 [cs].
Please convert into a bibtex citation.
[AI]:
bibtex @article{dao2022flashattention, title={FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness}, author={Dao, Tri and Fu, Daniel Y. and Ermon, Stefano and Rudra, Atri and R{\'e}, Christopher}, journal={arXiv preprint arXiv:2205.14135}, year={2022}, month={June}, url={http://arxiv.org/abs/2205.14135} }
[34] Baris E Suzek, Yuqi Wang, Hongzhan Huang, Peter B McGarvey, Cathy H Wu, and UniProt Consortium. UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches. Bioinformatics, 31(6):926-932, 2014. Publisher: Oxford University Press.
Please convert into a bibtex citation.
[AI]: Here's how the citation would look like in BibTeX format:
bibtex @article{Suzek2014, author = {Suzek, Baris E and Wang, Yuqi and Huang, Hongzhan and McGarvey, Peter B and Wu, Cathy H and UniProt Consortium}, title = {{UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches}}, journal = {Bioinformatics}, volume = {31}, number = {6}, pages = {926--932}, year = {2014}, publisher = {Oxford University Press}, }
[35] Lorna Richardson, Ben Allen, Germana Baldi, Martin Beracochea, Maxwell L Bileschi, Tony Burdett, Josephine Burgin, Juan Caballero-Pérez, Guy Cochrane, Lucy J Colwell, Tom Curtis, Alejandra Escobar-Zepeda, Tatiana A Gurbich, Varsha Kale, Anton Korobeynikov, Shriya Raj, Alexander B Rogers, Ekaterina Sakharova, Santiago Sanchez, Darren J Wilkinson, and Robert D Finn. MGnify: the microbiome sequence data analysis resource in 2023. Nucleic Acids Research, 51(D1): D753-D759, 12 2022. ISSN 0305-1048. doi: 10.1093/nar/gkac1080. URL https://doi.org/ $10.1093 / n a r / g k a c 1080$.
Please convert into a bibtex citation.
[AI]: Sure, here is the citation converted into a bibtex entry:
bibtex @article{Richardson2022MGnify, title={MGnify: the microbiome sequence data analysis resource in 2023}, author={Richardson, Lorna and Allen, Ben and Baldi, Germana and Beracochea, Martin and Bileschi, Maxwell L and Burdett, Tony and Burgin, Josephine and Caballero-Pérez, Juan and Cochrane, Guy and Colwell, Lucy J and Curtis, Tom and Escobar-Zepeda, Alejandra and Gurbich, Tatiana A and Kale, Varsha and Korobeynikov, Anton and Raj, Shriya and Rogers, Alexander B and Sakharova, Ekaterina and Sanchez, Santiago and Wilkinson, Darren J and Finn, Robert D}, journal={Nucleic Acids Research}, volume={51}, number={D1}, pages={D753-D759}, year={2022}, publisher={Oxford University Press}, issn = {0305-1048}, doi = {10.1093/nar/gkac1080}, url = {https://doi.org/10.1093/nar/gkac1080} }
Just ensure this matches the exact formatting and field requirements for your reference manager or document, as BibTeX syntax needs to be precise.
[36] Tobias H. Olsen, Fergus Boyles, and Charlotte M. Deane. Observed antibody space: A diverse database of cleaned, annotated, and translated unpaired and paired antibody sequences. Protein Science, 31 (1):141-146, 2022. doi: https://doi.org/10.1002/ pro.4205. URL https://onlinelibrary. wiley.com/doi/abs/10.1002/pro. 4205.
Please convert into a bibtex citation.
[AI]: Sure, here is the citation you provided converted into a BibTeX format:
bibtex @article{Olsen2022Observed, title={Observed antibody space: A diverse database of cleaned, annotated, and translated unpaired and paired antibody sequences}, author={Olsen, Tobias H. and Boyles, Fergus and Deane, Charlotte M.}, journal={Protein Science}, volume={31}, number={1}, pages={141--146}, year={2022}, publisher={Wiley Online Library}, doi={10.1002/pro.4205}, url={https://onlinelibrary.wiley.com/doi/abs/10.1002/pro.4205} }
Please ensure you replace any placeholder elements with the correct information for your particular need, and double-check for accuracy against your source.
[37] Stephen K Burley, Helen M Berman, Charmi Bhikadiya, Chunxiao Bi, Li Chen, Luigi Di Costanzo, Cole Christie, Ken Dalenberg, Jose M Duarte, Shuchismita Dutta, Zukang Feng, Sutapa Ghosh, David S Goodsell, Rachel K Green, Vladimir Guranoví, Dmytro Guzenko, Brian P Hudson, Tara Kalro, Yuhe Liang, Robert Lowe, Harry Namkoong, Ezra Peisach, Irina Periskova, Andreas Prlí, Chris Randle, Alexander Rose, Peter Rose, Raul Sala, Monica Sekharan, Chenghua Shao, Lihua Tan, Yi-Ping Tao, Yana Valasatava, Maria Voigt, John Westbrook, Jesse Woo, Huanwang Yang, Jasmine Young, Marina Zhuravleva, and Christine Zardecki. RCSB Protein Data Bank: biological macromolecular structures enabling research and education in fundamental biology, biomedicine, biotechnology and energy. Nucleic Acids Research, 47, 2019. doi: 10.1093/nar/gky1004. URL https: / / academic. oup.com/nar/article-abstract/47/ D1/D464/5144139.
Please convert into a bibtex citation.
[AI]: To convert the citation provided into a bibtex entry, you would use the following format:
bibtex @article{Burley2019, title={RCSB Protein Data Bank: biological macromolecular structures enabling research and education in fundamental biology, biomedicine, biotechnology and energy}, author={Burley, Stephen K and Berman, Helen M and Bhikadiya, Charmi and Bi, Chunxiao and Chen, Li and Di Costanzo, Luigi and Christie, Cole and Dalenberg, Ken and Duarte, Jose M and Dutta, Shuchismita and Feng, Zukang and Ghosh, Sutapa and Goodsell, David S and Green, Rachel K and Guranović, Vladimir and Guzenko, Dmytro and Hudson, Brian P and Kalro, Tara and Li, Yuhe and Lowe, Robert and Namkoong, Harry and Peisach, Ezra and Periskova, Irina and Prlí, Andreas and Randle, Chris and Rose, Alexander and Rose, Peter and Sala, Raul and Sekharan, Monica and Shao, Chenghua and Tan, Lihua and Tao, Yi-Ping and Valasatava, Yana and Voigt, Maria and Westbrook, John and Woo, Jesse and Yang, Huanwang and Young, Jasmine and Zhuravleva, Marina and Zardecki, Christine}, journal={Nucleic Acids Research}, volume={47}, year={2019}, doi={10.1093/nar/gky1004}, url={https://academic.oup.com/nar/article-abstract/47/D1/D464/5144139} }
This entry format is commonly used for citing journal articles in LaTeX documents using BibTeX. Adjustments can be made according to specific requirements or citation style guides if necessary.
[38] Typhaine Paysan-Lafosse, Matthias Blum, Sara Chuguransky, Tiago Grego, Beatriz Lázaro Pinto, Gustavo A Salazar, Maxwell L Bileschi, Peer Bork, Alan Bridge, Lucy Colwell, Julian Gough, Daniel H Haft, Ivica Letunić, Aron Marchler-Bauer, Huaiyu Mi, Darren A Natale, Christine A Orengo, Arun P Pandurangan, Catherine Rivoire, Christian J A Sigrist, Ian Sillitoe, Narmada Thanki, Paul D Thomas, Silvio C E Tosatto, Cathy H Wu, and Alex Bateman. InterPro in 2022. Nucleic Acids Research, 51(D1): D418-D427, January 2023. ISSN 0305-1048. doi: 10.1093/nar/gkac993. URL https://doi.org/ $10.1093 / n a r / g k a c 993$.
Please convert into a bibtex citation.
[AI]: Certainly, here is how you would format the provided citation into BibTeX format:
bibtex @article{PaysanLafosse2023, author = {Typhaine Paysan-Lafosse and Matthias Blum and Sara Chuguransky and Tiago Grego and Beatriz Lázaro Pinto and Gustavo A Salazar and Maxwell L Bileschi and Peer Bork and Alan Bridge and Lucy Colwell and Julian Gough and Daniel H Haft and Ivica Letunić and Aron Marchler-Bauer and Huaiyu Mi and Darren A Natale and Christine A Orengo and Arun P Pandurangan and Catherine Rivoire and Christian J A Sigrist and Ian Sillitoe and Narmada Thanki and Paul D Thomas and Silvio C E Tosatto and Cathy H Wu and Alex Bateman}, title = {InterPro in 2022}, journal = {Nucleic Acids Research}, volume = {51}, number = {D1}, pages = {D418-D427}, year = {2023}, issn = {0305-1048}, doi = {10.1093/nar/gkac993}, url = {https://doi.org/10.1093/nar/gkac993} }
Make sure to adjust the entry type (`@article` in this instance) and fields according to the requirements you have for your reference list, considering any specific citation style you're adhering to.
[39] Michel van Kempen, Stephanie Kim, Charlotte Tumescheit, Milot Mirdita, Johannes Söding, and Martin Steinegger. Foldseek: fast and accurate protein structure search. bioRxiv, February 2022. doi: 10.1101/2022.02.07.479398. URL http://biorxiv.org/lookup/doi/10. $1101 / 2022.02 .07 .479398$.
Please convert into a bibtex citation.
[AI]: Certainly, to convert the given citation into a Bibtex format, you can format it as follows:
bibtex @article{vanKempen2022Foldseek, title = {Foldseek: fast and accurate protein structure search}, author = {van Kempen, Michel and Kim, Stephanie and Tumescheit, Charlotte and Mirdita, Milot and Söding, Johannes and Steinegger, Martin}, journal = {bioRxiv}, year = {2022}, month = {Feb}, doi = {10.1101/2022.02.07.479398}, url = {http://biorxiv.org/lookup/doi/10.1101/2022.02.07.479398}, }
Please note that you should adjust the citation key (`vanKempen2022Foldseek`) based on your own citation management preferences. Additionally, the `month` field is given as text (i.e., "Feb"); for some bibliography styles or citation software, it might be preferable or required to use numerical month representation or full-month names. Ensure the entry meets the specific requirements of your bibliography manager or style guide.
[40] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, March 2022. URLhttp://arxiv.org/abs/2203.02155. arXiv:2203.02155 [cs].
Please convert into a bibtex citation.
[AI]: Here is how you can format the citation in BibTeX:
bibtex @article{ouyang2022training, title={Training language models to follow instructions with human feedback}, author={Ouyang, Long and Wu, Jeff and Jiang, Xu and Almeida, Diogo and Wainwright, Carroll L. and Mishkin, Pamela and Zhang, Chong and Agarwal, Sandhini and Slama, Katarina and Ray, Alex and Schulman, John and Hilton, Jacob and Kelton, Fraser and Miller, Luke and Simens, Maddie and Askell, Amanda and Welinder, Peter and Christiano, Paul and Leike, Jan and Lowe, Ryan}, journal={arXiv preprint arXiv:2203.02155}, year={2022}, month={mar}, url={http://arxiv.org/abs/2203.02155}, eprint={2203.02155}, archivePrefix={arXiv}, primaryClass={cs} }
[41] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model, December 2023. URL http://arxiv.org/abs/2305. 18290. arXiv:2305.18290 [cs].
Please convert into a bibtex citation.
[AI]:
bibtex @article{rafailov2023direct, title={Direct Preference Optimization: Your Language Model is Secretly a Reward Model}, author={Rafailov, Rafael and Sharma, Archit and Mitchell, Eric and Ermon, Stefano and Manning, Christopher D. and Finn, Chelsea}, journal={arXiv preprint arXiv:2305.18290}, year={2023}, month={December}, url={http://arxiv.org/abs/2305.18290} }
[42] Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative Reasoning Preference Optimization, May 2024. URL http://arxiv.org/abs/ 2404 . 19733. arXiv:2404.19733 [cs].
Please convert into a bibtex citation.
[AI]: Given the information, a BibTeX citation for this source could look like this:
bibtex @misc{pang2024iterative, author = {Pang, Richard Yuanzhe and Yuan, Weizie and Cho, Kyunghyun and He, He and Sukhbaatar, Sainbayar and Weston, Jason}, title = {Iterative Reasoning Preference Optimization}, year = {2024}, month = {May}, url = {http://arxiv.org/abs/2404.19733}, archivePrefix = {arXiv}, eprint = {2404.19733}, primaryClass = {cs} }
Note: For the future publication date, this entry assumes the paper was made available in May 2024 based on your input, even though current knowledge doesn't confirm this. Please adjust the date and any other details according to the actual publication information.
[43] Y. A. Labas, N. G. Gurskaya, Y. G. Yanushevich, A. F. Fradkov, K. A. Lukyanov, S. A. Lukyanov, and M. V. Matz. Diversity and evolution of the green fluorescent protein family. Proceedings of the National Academy of Sciences, 99 (7):4256-4261, April 2002. doi: 10.1073/pnas. 062552299. URL https://www.pnas.org/ doi/full/10.1073/pnas. 062552299 . Publisher: Proceedings of the National Academy of Sciences.
Please convert into a bibtex citation.
[AI]: Certainly! Here is the BibTeX citation for the given reference:
bibtex @article{Labas2002DiversityAE, title={Diversity and evolution of the green fluorescent protein family}, author={Y. A. Labas and N. G. Gurskaya and Y. G. Yanushevich and A. F. Fradkov and K. A. Lukyanov and S. A. Lukyanov and M. V. Matz}, journal={Proceedings of the National Academy of Sciences}, volume={99}, number={7}, pages={4256--4261}, year={2002}, month={Apr}, publisher={Proceedings of the National Academy of Sciences}, doi={10.1073/pnas.062552299}, url={https://www.pnas.org/content/99/7/4256} }
This BibTeX citation includes elements such as the title of the article, authors, journal name, volume, number, pages, year, month, publisher, DOI, and URL. Make sure to adjust this entry to match the specific formatting and field requirements of your citation manager or document preparation system.
[44] Louisa Gonzalez Somermeyer, Aubin Fleiss, Alexander S Mishin, Nina G Bozhanova, Anna A Igolkina, Jens Meiler, Maria-Elisenda Alaball Pujol, Ekaterina V Putintseva, Karen S Sarkisyan, and Fyodor A Kondrashov. Heterogeneity of the GFP fitness landscape and data-driven protein design. eLife, 11: e75842, May 2022. ISSN 2050-084X. doi: 10.7554/ eLife.75842. URL https://www.ncbi.nlm. nih.gov/pmc/articles/PMC9119679/.
Please convert into a bibtex citation.
[AI]:
bibtex @article{GonzalezSomermeyer2022, title={Heterogeneity of the GFP fitness landscape and data-driven protein design}, author={Gonzalez Somermeyer, Louisa and Fleiss, Aubin and Mishin, Alexander S and Bozhanova, Nina G and Igolkina, Anna A and Meiler, Jens and Alaball Pujol, Maria-Elisenda and Putintseva, Ekaterina V and Sarkisyan, Karen S and Kondrashov, Fyodor A}, journal={eLife}, volume={11}, pages={e75842}, year={2022}, month={May}, publisher={eLife Sciences Publications Ltd}, doi={10.7554/eLife.75842}, ISSN={2050-084X}, URL={https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9119679/} }
[45] Karen S. Sarkisyan, Dmitry A. Bolotin, Margarita V. Meer, Dinara R. Usmanova, Alexander S. Mishin, George V. Sharonov, Dmitry N. Ivankov, Nina G. Bozhanova, Mikhail S. Baranov, Onuralp Soylemez, Natalya S. Bogatyreva, Peter K. Vlasov, Evgeny S. Egorov, Maria D. Logacheva, Alexey S. Kondrashov, Dmitry M. Chudakov, Ekaterina V. Putintseva, Ilgar Z. Mamedov, Dan S. Tawfik, Konstantin A. Lukyanov, and Fyodor A. Kondrashov. Local fitness landscape of the green fluorescent protein. Nature, 533(7603):397-401, May 2016. ISSN 14764687. doi: 10.1038/nature17995. URL https://www. nature.com/articles/nature17995. Publisher: Nature Publishing Group.
Please convert into a bibtex citation.
[AI]:
bibtex @article{Sarkisyan2016, title={Local fitness landscape of the green fluorescent protein}, author={Sarkisyan, Karen S. and Bolotin, Dmitry A. and Meer, Margarita V. and Usmanova, Dinara R. and Mishin, Alexander S. and Sharonov, George V. and Ivankov, Dmitry N. and Bozhanova, Nina G. and Baranov, Mikhail S. and Soylemez, Onuralp and Bogatyreva, Natalya S. and Vlasov, Peter K. and Egorov, Evgeny S. and Logacheva, Maria D. and Kondrashov, Alexey S. and Chudakov, Dmitry M. and Putintseva, Ekaterina V. and Mamedov, Ilgar Z. and Tawfik, Dan S. and Lukyanov, Konstantin A. and Kondrashov, Fyodor A.}, journal={Nature}, volume={533}, number={7603}, pages={397--401}, year={2016}, publisher={Nature Publishing Group}, doi={10.1038/nature17995}, issn={14764687}, url={https://www.nature.com/articles/nature17995} }
[46] Jonathan Yaacov Weinstein, Carlos Martí-Gómez, Rosalie Lipsh-Sokolik, Shlomo Yakir Hoch, Demian Liebermann, Reinat Nevo, Haim Weissman, Ekaterina Petrovich-Kopitman, David Margulies, Dmitry Ivankov, David M. McCandlish, and Sarel J. Fleishman. Designed active-site library reveals thousands of functional GFP variants. Nature Communications, 14(1):2890, May 2023. ISSN 20411723. doi: 10.1038/s41467-023-38099-z. URL https://www.nature.com/articles/ s41467-023-38099-z. Publisher: Nature Publishing Group.
Please convert into a bibtex citation.
[AI]:
bibtex @article{Weinstein2023, title={Designed active-site library reveals thousands of functional GFP variants}, author={Weinstein, Jonathan Yaacov and Martí-Gómez, Carlos and Lipsh-Sokolik, Rosalie and Hoch, Shlomo Yakir and Liebermann, Demian and Nevo, Reinat and Weissman, Haim and Petrovich-Kopitman, Ekaterina and Margulies, David and Ivankov, Dmitry and McCandlish, David M. and Fleishman, Sarel J.}, journal={Nature Communications}, volume={14}, number={1}, pages={2890}, year={2023}, month={May}, publisher={Nature Publishing Group}, doi={10.1038/s41467-023-38099-z}, ISSN={20411723}, url={https://www.nature.com/articles/s41467-023-38099-z} }
[47] Surojit Biswas, Gleb Kuznetsov, Pierce J Ogden, Nicholas J Conway, Ryan P Adams, and George M Church. Toward machine-guided design of proteins. bioRxiv, page 337154, 2018. doi: 10.1101/ 337154. URL https://www.biorxiv.org/ content/early/2018/06/02/337154.
Please convert into a bibtex citation.
[AI]: Here is how you can format the citation as a BibTeX entry:
bibtex @article{Biswas2018MachineGuided, title={Toward machine-guided design of proteins}, author={Surojit Biswas and Gleb Kuznetsov and Pierce J Ogden and Nicholas J Conway and Ryan P Adams and George M Church}, journal={bioRxiv}, pages={337154}, year={2018}, doi={10.1101/337154}, url={https://www.biorxiv.org/content/early/2018/06/02/337154}, }
Please, make sure to double-check this entry for accuracy and completeness, especially considering the dynamic nature of pre-print repositories like bioRxiv and the possibility of future updates to the document or metadata.
[48] Surojit Biswas, Grigory Khimulya, Ethan C Alley, Kevin M Esvelt, and George M Church. Low-n protein engineering with data-efficient deep learning. Nature methods, 18(4):389-396, 2021.
Please convert into a bibtex citation.
[AI]:
bibtex @article{Biswas2021LowNProtein, title={Low-n protein engineering with data-efficient deep learning}, author={Biswas, Surojit and Khimulya, Grigory and Alley, Ethan C and Esvelt, Kevin M and Church, George M}, journal={Nature methods}, volume={18}, number={4}, pages={389--396}, year={2021}, publisher={Nature Publishing Group} }
[49] Mats Ormö, Andrew B. Cubitt, Karen Kallio, Larry A. Gross, Roger Y. Tsien, and S. James Remington. Crystal Structure of the Aequorea victoria Green Fluorescent Protein. Science, $\quad 273(5280): 1392-1395, \quad$ September 1996. doi: 10.1126/science.273.5280.1392. URL https://www.science.org/doi/10. 1126/science.273.5280.1392. Publisher: American Association for the Advancement of Science.
Please convert into a bibtex citation.
[AI]:
bibtex @article{Ormo1996, title={Crystal Structure of the Aequorea victoria Green Fluorescent Protein}, author={Orm{\"o}, Mats and Cubitt, Andrew B. and Kallio, Karen and Gross, Larry A. and Tsien, Roger Y. and Remington, S. James}, journal={Science}, volume={273}, number={5280}, pages={1392--1395}, year={1996}, month={Sep}, publisher={American Association for the Advancement of Science}, doi={10.1126/science.273.5280.1392}, url={https://www.science.org/doi/10.1126/science.273.5280.1392} }
[50] David P. Barondeau, Christopher D. Putnam, Carey J. Kassmann, John A. Tainer, and Elizabeth D. Getzoff. Mechanism and energetics of green fluorescent protein chromophore synthesis revealed by trapped intermediate structures. Proceedings of the National
Academy of Sciences, 100(21):12111-12116, October 2003. doi: 10.1073/pnas.2133463100. URL https://www.pnas.org/doi/full/ 10.1073/pnas.2133463100. Publisher: Proceedings of the National Academy of Sciences.
Please convert into a bibtex citation.
[AI]:
bibtex @article{Barondeau2003, title = {Mechanism and energetics of green fluorescent protein chromophore synthesis revealed by trapped intermediate structures}, author = {Barondeau, David P. and Putnam, Christopher D. and Kassmann, Carey J. and Tainer, John A. and Getzoff, Elizabeth D.}, journal = {Proceedings of the National Academy of Sciences}, volume = {100}, number = {21}, pages = {12111--12116}, year = {2003}, doi = {10.1073/pnas.2133463100}, url = {https://www.pnas.org/doi/full/10.1073/pnas.2133463100}, publisher = {Proceedings of the National Academy of Sciences}, month = oct }
[51] Christiam Camacho, George Coulouris, Vahram Avagyan, Ning Ma, Jason Papadopoulos, Kevin Bealer, and Thomas L Madden. Blast+: architecture and applications. BMC bioinformatics, 10:1-9, 2009.
Please convert into a bibtex citation.
[AI]:
bibtex @article{Camacho2009BLAST, title={BLAST+: architecture and applications}, author={Christiam Camacho and George Coulouris and Vahram Avagyan and Ning Ma and Jason Papadopoulos and Kevin Bealer and Thomas L Madden}, journal={BMC Bioinformatics}, volume={10}, pages={1-9}, year={2009}, publisher={BioMed Central} }
[52] Martin Steinegger and Johannes Söding. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology, 35(11):1026-1028, 2017.
Please convert into a bibtex citation.
[AI]:
bibtex @article{Steinegger2017MMseqs2, title={MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets}, author={Steinegger, Martin and Söding, Johannes}, journal={Nature biotechnology}, volume={35}, number={11}, pages={1026--1028}, year={2017}, publisher={Nature Publishing Group} }
[53] Andrea M. Quattrini, Estefanía Rodríguez, Brant C. Faircloth, Peter F. Cowman, Mercer R. Brugler, Gabriela A. Farfan, Michael E. Hellberg, Marcelo V. Kitahara, Cheryl L. Morrison, David A. Paz-García, James D. Reimer, and Catherine S. McFadden. Palaeoclimate ocean conditions shaped the evolution of corals and their skeletons through deep time. Nature Ecology \& Evolution, 4(11):1531-1538, August 2020. ISSN 2397334X. doi: 10.1038/s41559-020-01291-1. URL https://www.nature.com/articles/ s41559-020-01291-1.
Please convert into a bibtex citation.
[AI]:
bibtex @article{Quattrini2020, title={Palaeoclimate ocean conditions shaped the evolution of corals and their skeletons through deep time}, author={Quattrini, Andrea M. and Rodríguez, Estefanía and Faircloth, Brant C. and Cowman, Peter F. and Brugler, Mercer R. and Farfan, Gabriela A. and Hellberg, Michael E. and Kitahara, Marcelo V. and Morrison, Cheryl L. and Paz-García, David A. and Reimer, James D. and McFadden, Catherine S.}, journal={Nature Ecology \& Evolution}, volume={4}, number={11}, pages={1531--1538}, year={2020}, publisher={Nature Publishing Group}, ISSN={2397334X}, doi={10.1038/s41559-020-01291-1}, url={https://www.nature.com/articles/s41559-020-01291-1} }
[54] John Maynard Smith. Natural selection and the concept of a protein space. Nature, 225(5232):563-564, 1970 .
Please convert into a bibtex citation.
[AI]:
bibtex @article{Smith1970, author = {Smith, John Maynard}, title = {Natural selection and the concept of a protein space}, journal = {Nature}, volume = {225}, number = {5232}, pages = {563--564}, year = {1970} }
[55] Geoffrey E. Hinton, James L. McClelland, and David E. Rumelhart. Distributed representations. In The Philosophy of Artificial Intelligence, 1986.
Please convert into a bibtex citation.
[AI]:
bibtex @incollection{Hinton1986Distributed, title={Distributed representations}, author={Hinton, Geoffrey E. and McClelland, James L. and Rumelhart, David E.}, booktitle={The Philosophy of Artificial Intelligence}, year={1986}, editor={}, pages={}, publisher={}, address={} }
**Note**: Since the full details of the editor, pages, publisher, and address were not provided in your original query, I left these fields empty. You may want to fill in these details if you have access to the full reference.
[56] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 1999.
Please convert into a bibtex citation.
[AI]:
bibtex @article{tishby1999information, title={The information bottleneck method}, author={Tishby, Naftali and Pereira, Fernando C and Bialek, William}, journal={arXiv preprint physics/0004057}, year={1999} }
[57] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Advances in Neural Information Processing Systems, pages 5998-6008, 2017. URL https://papers.nips.cc/paper/ 7181-attention-is-all-you-need. $\mathrm{pdf}$.
Please convert into a bibtex citation.
[AI]:
bibtex @inproceedings{Vaswani2017AttentionIA, title={Attention Is All You Need}, author={Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and {\L}ukasz Kaiser and Illia Polosukhin}, booktitle={Advances in Neural Information Processing Systems}, pages={5998--6008}, year={2017}, url={https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf} }
[58] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. On layer normalization in the transformer architecture. arXiv:2002.04745, 2020.
Please convert into a bibtex citation.
[AI]: Here is the BibTeX citation for the mentioned article:
bibtex @misc{Xiong2020OnLayer, title={On Layer Normalization in the Transformer Architecture}, author={Ruibin Xiong and Yunchang Yang and Di He and Kai Zheng and Shuxin Zheng and Chen Xing and Huishuai Zhang and Yanyan Lan and Liwei Wang and Tie-Yan Liu}, year={2020}, eprint={2002.04745}, archivePrefix={arXiv}, primaryClass={cs.LG} }
Please, note that this citation is formatted following general guidelines for citing an arXiv preprint. Depending on your specific requirements or citation style you might need to adjust it. For example, the field `misc` could be replaced with `article` if you are using a citation style that does not recognize the `misc` type, or if `article` is more appropriate for the context in which you're citing the paper. Additionally, ensure that the `eprint`, `archivePrefix`, and `primaryClass` fields are acceptable in your bibliography's context or if they should be omitted for simplicity or style compliance.
[59] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873):583-589, August 2021. ISSN 14764687. doi: 10.1038/s41586-021-03819-2. URL https://www.nature.com/articles/ s41586-021-03819-2. Bandiera_abtest: a Cc_license_type: cc_by Cg_type: Nature Research Journals Number: 7873 Primary_atype: Research Publisher: Nature Publishing Group Subject_term: Computational biophysics;Machine learning;Protein structure predictions;Structural biology Subject_term_id: computational-biophysics;machinelearning;protein-structure-predictions;structuralbiology.
Please convert into a bibtex citation.
[AI]:
bibtex @article{Jumper2021HighlyAP, title={Highly accurate protein structure prediction with AlphaFold}, author={John Jumper and Richard Evans and Alexander Pritzel and Tim Green and Michael Figurnov and Olaf Ronneberger and Kathryn Tunyasuvunakool and Russ Bates and Augustin {\v{Z}}{\'i}dek and Anna Potapenko and Alex Bridgland and Clemens Meyer and Simon A. A. Kohl and Andrew J. Ballard and Andrew Cowie and Bernardino Romera-Paredes and Stanislav Nikolov and Rishub Jain and Jonas Adler and Trevor Back and Stig Petersen and David Reiman and Ellen Clancy and Michal Zielinski and Martin Steinegger and Michalina Pacholska and Tamas Berghammer and Sebastian Bodenstein and David Silver and Oriol Vinyals and Andrew W. Senior and Koray Kavukcuoglu and Pushmeet Kohli and Demis Hassabis}, journal={Nature}, volume={596}, number={7873}, pages={583-589}, year={2021}, publisher={Nature Publishing Group}, doi={10.1038/s41586-021-03819-2}, url={https://www.nature.com/articles/s41586-021-03819-2}, ISSN={14764687}, cclicensetype={ccby}, cgtype={Nature Research Journals}, primaryatype={Research}, subjectterm={Computational biophysics;Machine learning;Protein structure predictions;Structural biology}, subjecttermid={computational-biophysics;machinelearning;protein-structure-predictions;structuralbiology} }
[60] Wolfgang Kabsch and Christian Sander. Dictionary of protein secondary structure: Pattern recognition of hydrogen-bonded and geometrical features. Biopolymers: Original Research on Biomolecules, 1983.
Please convert into a bibtex citation.
[AI]:
bibtex @article{KabschSander1983, title={Dictionary of protein secondary structure: Pattern recognition of hydrogen-bonded and geometrical features}, author={Kabsch, Wolfgang and Sander, Christian}, journal={Biopolymers: Original Research on Biomolecules}, volume={22}, number={12}, pages={2577--2637}, year={1983}, publisher={Wiley Subscription Services, Inc., A Wiley Company} }
[61] Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced Transformer with Rotary Position Embedding, October 2021. URL http://arxiv.org/abs/2104.09864. arXiv:2104.09864 [cs] version: 2.
Please convert into a bibtex citation.
[AI]: Here is the requested citation in BibTeX format:
bibtex @article{su2021roformer, title={RoFormer: Enhanced Transformer with Rotary Position Embedding}, author={Su, Jianlin and Lu, Yu and Pan, Shengfeng and Wen, Bo and Liu, Yunfeng}, journal={arXiv preprint arXiv:2104.09864}, year={2021}, month={Oct}, version={2}, url={http://arxiv.org/abs/2104.09864}, archivePrefix={arXiv}, primaryClass={cs.CL} }
Please note that I set the primary class to `cs.CL` (Computation and Language) which is often used for papers related to natural language processing, assuming from the context of the paper title. If the paper belongs to a different category within computer science, you may need to adjust the `primaryClass` field accordingly.
[62] Noam Shazeer. GLU Variants Improve Transformer, February 2020. URL http: / / arxiv. org/abs / 2002.05202. arXiv:2002.05202 [cs, stat].
Please convert into a bibtex citation.
[AI]: To cite the article titled "GLU Variants Improve Transformer" by Noam Shazeer in BibTeX format, you would format the entry as follows:
bibtex @misc{shazeer2020glu, title={GLU Variants Improve Transformer}, author={Shazeer, Noam}, year={2020}, month={Feb}, howpublished={\url{http://arxiv.org/abs/2002.05202}}, eprint={2002.05202}, archivePrefix={arXiv}, primaryClass={cs, stat} }
Please note:
- The entry type is set as `@misc` since this is a preprint not yet published in a formal peer-reviewed conference or journal at the time of this citation.
- The `eprint`, `archivePrefix`, and `primaryClass` fields are included for citing arXiv preprints specifically, providing the article's unique identifier, the archive prefix, and the primary classification (in this case, `cs, stat` indicating computer science and statistics).
- The `year` and `month` reflect the preprint's posted date.
- Always check the latest guidance or requirements from your institution or publisher for citation formats, as there may be specific preferences or variations in how elements are represented in BibTeX entries.
[63] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov,
Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling Language Modeling with Pathways, April 2022. URLhttp://arxiv.org/abs/2204.02311. arXiv:2204.02311 [cs].
Please convert into a bibtex citation.
[AI]:
bibtex @misc{chowdhery2022palm, title={PaLM: Scaling Language Modeling with Pathways}, author={Aakanksha Chowdhery and Sharan Narang and Jacob Devlin and Maarten Bosma and Gaurav Mishra and Adam Roberts and Paul Barham and Hyung Won Chung and Charles Sutton and Sebastian Gehrmann and Parker Schuh and Kensen Shi and Sasha Tsvyashchenko and Joshua Maynez and Abhishek Rao and Parker Barnes and Yi Tay and Noam Shazeer and Vinodkumar Prabhakaran and Emily Reif and Nan Du and Ben Hutchinson and Reiner Pope and James Bradbury and Jacob Austin and Michael Isard and Guy Gur-Ari and Pengcheng Yin and Toju Duke and Anselm Levskaya and Sanjay Ghemawat and Sunipa Dev and Henryk Michalew and Xavier Garcia and Vedant Misra and Kevin Robinson and Liam Fedus and Denny Zhou and Daphne Ippolito and David Luan and Hyeontaek Lim and Barret Zoph and Alexander Spiridonov and Ryan Sepassi and David Dohan and Shivani Agrawal and Mark Omernick and Andrew M. Dai and Thanumalayan Sankaranarayana Pillai and Marie Pellat and Aitor Lewkowycz and Erica Moreira and Rewon Child and Oleksandr Polozov and Katherine Lee and Zongwei Zhou and Xuezhi Wang and Brennan Saeta and Mark Diaz and Orhan Firat and Michele Catasta and Jason Wei and Kathy Meier-Hellstern and Douglas Eck and Jeff Dean and Slav Petrov and Noah Fiedel}, year={2022}, eprint={2204.02311}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={http://arxiv.org/abs/2204.02311} }
[64] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling Laws for Autoregressive Generative Modeling. CoRR, abs/2010.14701, 2020. URL https://arxiv.org/abs/2010. 14701. _eprint: 2010.14701.
Please convert into a bibtex citation.
[AI]: Here is how the citation would look like in BibTeX format:
bibtex @article{henighan2020scaling, title={Scaling Laws for Autoregressive Generative Modeling}, author={Henighan, Tom and Kaplan, Jared and Katz, Mor and Chen, Mark and Hesse, Christopher and Jackson, Jacob and Jun, Heewoo and Brown, Tom B. and Dhariwal, Prafulla and Gray, Scott and Hallacy, Chris and Mann, Benjamin and Radford, Alec and Ramesh, Aditya and Ryder, Nick and Ziegler, Daniel M. and Schulman, John and Amodei, Dario and McCandlish, Sam}, journal={CoRR}, volume={abs/2010.14701}, year={2020}, url={https://arxiv.org/abs/2010.14701}, eprint={2010.14701} }
Please remember to always check the generated citation for correctness and make sure it fits the style-guide requirements of your respective academic work.
[65] Noam Wies, Yoav Levine, Daniel Jannai, and Amnon Shashua. Which transformer architecture fits my data? a vocabulary bottleneck in self-attention, 2021.
Please convert into a bibtex citation.
[AI]:
bibtex @article{Wies2021WhichTA, title={Which transformer architecture fits my data? A vocabulary bottleneck in self-attention}, author={Noam Wies and Yoav Levine and Daniel Jannai and Amnon Shashua}, journal={arXiv preprint arXiv:2109.04504}, year={2021} }
[66] John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative Models for Graph-Based Protein Design. page 12, 2019. URL https://papers.nips.cc/paper/
Please convert into a bibtex citation.
[AI]: Here is how you can convert the provided reference into a BibTeX citation format:
bibtex @inproceedings{ingraham2019generative, title={Generative Models for Graph-Based Protein Design}, author={Ingraham, John and Garg, Vikas and Barzilay, Regina and Jaakkola, Tommi}, booktitle={Advances in Neural Information Processing Systems}, pages={12}, year={2019}, organization={Curran Associates, Inc.}, url={https://papers.nips.cc/paper/} }
Please note:
- The `booktitle` field is set to "Advances in Neural Information Processing Systems" as the URL suggests it's from the NeurIPS conference, but the exact volume or conference number isn't provided based on the initial reference.
- The `pages` field is populated with "12" as provided, but typically, this would denote the starting page or a page range if available.
- The `url` field is pointing to the main page for NeurIPS papers rather than the direct URL to the paper because a specific URL was not provided. You might need to search for the paper on the NeurIPS website and update this field with the direct link to the paper for more precise referencing.
- The `organization` field is assumed to be "Curran Associates, Inc." because they are the publisher for many conference proceedings, including NeurIPS, but this might need updating based on the actual publication details.
9711-generative-models-for-graph-based-protein
[67] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural Discrete Representation Learning. arXiv:1711.00937 [cs], May 2018. URLhttp://arxiv.org/abs/1711.00937. arXiv: 1711.00937.
Please convert into a bibtex citation.
[AI]: Certainly! Here's how you could format that citation in BibTeX:
bibtex @article{vandenOord2018NeuralDiscrete, title={Neural Discrete Representation Learning}, author={van den Oord, Aaron and Vinyals, Oriol and Kavukcuoglu, Koray}, journal={arXiv preprint arXiv:1711.00937}, year={2018}, month={May}, url={http://arxiv.org/abs/1711.00937}, }
This entry is tailored for the preprint version of the paper on arXiv. Depending on your citation manager or LaTeX editor, you might need to adjust the fields slightly, but this should work for most standard BibTeX setups.
[68] Ali Razavi, Aäron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQVAE-2. CoRR, abs/1906.00446, 2019. URL http: //arxiv.org/abs/1906.00446.
Please convert into a bibtex citation.
[AI]:
bibtex @article{Razavi2019VQVAE2, title={Generating diverse high-fidelity images with VQVAE-2}, author={Razavi, Ali and van den Oord, Aäron and Vinyals, Oriol}, journal={CoRR}, volume={abs/1906.00446}, year={2019}, url={http://arxiv.org/abs/1906.00446} }
[69] Aurko Roy, Ashish Vaswani, Arvind Neelakantan, and Niki Parmar. Theory and experiments on vector quantized autoencoders. CoRR, abs/1805.11063, 2018. URL http://arxiv.org/abs/1805. 11063 .
Please convert into a bibtex citation.
[AI]:
bibtex @article{Roy2018TheoryAE, title={Theory and experiments on vector quantized autoencoders}, author={Aurko Roy and Ashish Vaswani and Arvind Neelakantan and Niki Parmar}, journal={CoRR}, volume={abs/1805.11063}, year={2018}, url={http://arxiv.org/abs/1805.11063} }
[70] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-toimage generation, 2022.
Please convert into a bibtex citation.
[AI]: ```
@article{yu2022scaling,
title={Scaling autoregressive models for content-rich text-to-image generation},
author={Yu, Jiahui and Xu, Yuanzhong and Koh, Jing Yu and Luong, Thang and Baid, Gunjan and Wang, Zirui and Vasudevan, Vijay and Ku, Alexander and Yang, Yinfei and Ayan, Burcu Karagol and Hutchinson, Ben and Han, Wei and Parekh, Zarana and Li, Xin and Zhang, Han and Baldridge, Jason and Wu, Yonghui},
journal={},
volume={},
number={},
pages={},
year={2022},
publisher={}
}
Please note that I left the fields for journal, volume, number, pages, and publisher empty because these details are not provided in the provided reference. You may need to fill these in with the correct information depending on where the work was published or presented.
[71] The UniProt Consortium. UniProt: the Universal Protein Knowledgebase in 2023. Nucleic Acids Research, 51(D1):D523-D531, 11 2022. ISSN 03051048. doi: 10.1093/nar/gkac1052. URL https: //doi.org/10.1093/nar/gkac1052. Please convert into a bibtex citation.
AI:
@article{UniProt2023,
title={UniProt: the Universal Protein Knowledgebase in 2023},
author={The UniProt Consortium},
journal={Nucleic Acids Research},
volume={51},
number={D1},
pages={D523--D531},
year={2022},
month={11},
issn={03051048},
doi={10.1093/nar/gkac1052},
url={https://doi.org/10.1093/nar/gkac1052}
}
[72] I-Min A Chen, Ken Chu, Krishnaveni Palaniappan, Anna Ratner, Jinghua Huang, Marcel Huntemann, Patrick Hajek, Stephan J Ritter, Cody Webb, Dongying Wu, Neha J Varghese, T B K Reddy, Supratim Mukherjee, Galina Ovchinnikova, Matt Nolan, Rekha Seshadri, Simon Roux, Axel Visel, Tanja Woyke, Emiley A Eloe-Fadrosh, Nikos C Kyrpides, and Natalia N Ivanova. The IMG/M data management and analysis system v.7: content updates and new features. Nucleic Acids Research, 51 (D1):D723-D732, 11 2022. ISSN 0305-1048. doi: 10.1093/nar/gkac976. URL https: / doi.org/ $10.1093 /$ nar/gkac976. Please convert into a bibtex citation.
AI:
@article{Chen2022IMG,
author = {Chen, I-Min A and Chu, Ken and Palaniappan, Krishnaveni and Ratner, Anna and Huang, Jinghua and Huntemann, Marcel and Hajek, Patrick and Ritter, Stephan J and Webb, Cody and Wu, Dongying and Varghese, Neha J and Reddy, T B K and Mukherjee, Supratim and Ovchinnikova, Galina and Nolan, Matt and Seshadri, Rekha and Roux, Simon and Visel, Axel and Woyke, Tanja and Eloe-Fadrosh, Emiley A and Kyrpides, Nikos C and Ivanova, Natalia N},
title = {The IMG/M data management and analysis system v.7: content updates and new features},
journal = {Nucleic Acids Research},
volume = {51},
number = {D1},
pages = {D723-D732},
year = {2022},
doi = {10.1093/nar/gkac976},
ISSN = {0305-1048},
url = {https://doi.org/10.1093/nar/gkac976},
month = {11}
}
[73] Martin Steinegger and Johannes Söding. MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature Biotechnology, 35(11):1026-1028, November 2017. ISSN 15461696. doi: 10.1038/nbt.3988. URL https: / /www . nature.com/articles/nbt.3988. Number: 11 Publisher: Nature Publishing Group. Please convert into a bibtex citation.
AI:
@article{Steinegger2017,
author = {Steinegger, Martin and S{\"o}ding, Johannes},
title = {MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets},
journal = {Nature Biotechnology},
volume = {35},
number = {11},
pages = {1026--1028},
year = {2017},
doi = {10.1038/nbt.3988},
ISSN = {15461696},
url = {https://www.nature.com/articles/nbt.3988},
publisher = {Nature Publishing Group}
}
[74] Philip Jones, David Binns, Hsin-Yu Chang, Matthew Fraser, Weizhong Li, Craig McAnulla, Hamish McWilliam, John Maslen, Alex Mitchell, Gift Nuka, Sebastien Pesseat, Antony F. Quinn, Amaia Sangrador-Vegas, Maxim Scheremetjew, Siew-Yit Yong, Rodrigo Lopez, and Sarah Hunter. InterProScan 5: genome-scale protein function classification. Bioinformatics, 30(9):1236-1240, 012014. ISSN 1367-4803. doi: 10.1093/bioinformatics/ btu031. URL https://doi.org/10.1093/ bioinformatics/btu031. Please convert into a bibtex citation.
AI:
@article{Jones2014InterProScan5,
title={InterProScan 5: genome-scale protein function classification},
author={Jones, Philip and Binns, David and Chang, Hsin-Yu and Fraser, Matthew and Li, Weizhong and McAnulla, Craig and McWilliam, Hamish and Maslen, John and Mitchell, Alex and Nuka, Gift and Pesseat, Sebastien and Quinn, Antony F. and Sangrador-Vegas, Amaia and Scheremetjew, Maxim and Yong, Siew-Yit and Lopez, Rodrigo and Hunter, Sarah},
journal={Bioinformatics},
volume={30},
number={9},
pages={1236--1240},
year={2014},
publisher={Oxford University Press},
issn={1367-4803},
doi={10.1093/bioinformatics/btu031},
url={https://doi.org/10.1093/bioinformatics/btu031}
}
[75] Patrick Kunzmann and Kay Hamacher. Biotite: a unifying open source computational biology framework in Python. BMC Bioinformatics, 19(1):346, October 2018. ISSN 1471-2105. doi: 10.1186/ s12859-018-2367-z. URL https://doi.org/ $10.1186 / s 12859-018-2367-z$. Please convert into a bibtex citation.
AI:
@article{Kunzmann2018,
title = {Biotite: A unifying open source computational biology framework in Python},
author = {Kunzmann, Patrick and Hamacher, Kay},
journal = {BMC Bioinformatics},
volume = {19},
number = {1},
pages = {346},
year = {2018},
month = {Oct},
issn = {1471-2105},
doi = {10.1186/s12859-018-2367-z},
url = {https://doi.org/10.1186/s12859-018-2367-z}
}
[76] Wouter G. Touw, Coos Baakman, Jon Black, Tim A. H. te Beek, E. Krieger, Robbie P. Joosten, and Gert Vriend. A series of PDB-related databanks for everyday needs. Nucleic Acids Research, 43(D1):D364-D368, January 2015. ISSN 03051048. doi: 10.1093/nar/gku1028. URL https: //doi.org/10.1093/nar/gku1028. Please convert into a bibtex citation.
AI:
@article{Touw2015,
author = {Touw, Wouter G. and Baakman, Coos and Black, Jon and te Beek, Tim A. H. and Krieger, E. and Joosten, Robbie P. and Vriend, Gert},
title = {A series of PDB-related databanks for everyday needs},
journal = {Nucleic Acids Research},
volume = {43},
number = {D1},
pages = {D364-D368},
year = {2015},
doi = {10.1093/nar/gku1028},
ISSN = {03051048},
url = {https://doi.org/10.1093/nar/gku1028},
month = jan
}
[77] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv:1711.05101, 2017. [78] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fsdp: Experiences on scaling fully sharded data parallel, 2023. Please convert into a bibtex citation.
AI: Certainly! Here are the BibTeX citations for both references:
@article{LoshchilovHutter2017,
title={{Decoupled weight decay regularization}},
author={Loshchilov, Ilya and Hutter, Frank},
journal={arXiv:1711.05101},
year={2017},
}
For the second reference, it's slightly trickier because it likely hasn't been published in a journal or conference yet (given it cites a year but no specific journal/conference), and the citation is based on the title mentioning PyTorch FSDP, authors, and a year. Here's a close approximation:
@article{ZhaoPyTorchFSDP2023,
title={{PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel}},
author={Zhao, Yanli and Gu, Andrew and Varma, Rohan and Luo, Liang and Huang, Chien-Chin and Xu, Min and Wright, Less and Shojanazeri, Hamid and Ott, Myle and Shleifer, Sam and Desmaison, Alban and Balioglu, Can and Damania, Pritam and Nguyen, Bernard and Chauhan, Geeta and Hao, Yuchen and Mathews, Ajit and Li, Shen},
year={2023},
note = {arXiv preprint arXiv:SpecificIDIfNeeded}
}
Because the second reference does not specify the journal or arXiv ID, you might need to add journal={} or replace the note field with journal={arXiv preprint arXiv:YourSpecificID} if you have the specific arXiv ID.
[79] NVIDIA. Transformer engine. https://github. com/NVIDIA/TransformerEngine, 2024. Please convert into a bibtex citation.
AI: Given the original citation you provided and adjusting to fit a standard BibTeX format (while also considering that the year you provided is in the future and likely a typo or incorrect), the citation might look something like this if converted to BibTeX:
@misc{NVIDIA2024,
author = {{NVIDIA}},
title = {Transformer Engine},
year = {2024},
howpublished = {\url{https://github.com/NVIDIA/TransformerEngine}},
note = {Accessed: 2023-09-30}
}
However, please adjust the year (2024) if it was mentioned in error. If you are truly citing a future or expected publication/release, ensure that this citation is updated once the correct details are available. If your citation is supposed to be for material available/read in 2023 or prior, please correct the year accordingly.
Remember to replace "Accessed: 2023-09-30" with the actual date you accessed the resource. Additionally, while this entry assumes a @misc type due to its general nature and online accessibility, you may adjust the entry type based on the specific content and context within your bibliography (for example, using @online if your bibliography style supports it).
[80] Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hackable transformer modelling library. https://github.com/ facebookresearch/xformers, 2022. Please convert into a bibtex citation.
AI:
@misc{lefaudeux2022xformers,
title = {{xformers: A modular and hackable transformer modelling library}},
author = {Benjamin Lefaudeux and Francisco Massa and Diana Liskovich and Wenhan Xiong and Vittorio Caggiano and Sean Naren and Min Xu and Jieru Hu and Marta Tintore and Susan Zhang and Patrick Labatut and Daniel Haziza and Luca Wehrstedt and Jeremy Reizenstein and Grigory Sizov},
year = {2022},
howpublished = {\url{https://github.com/facebookresearch/xformers}},
}
[81] Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth, 2023. Please convert into a bibtex citation.
AI:
@article{Dong2023AttentionIN,
title={Attention is not all you need: Pure attention loses rank doubly exponentially with depth},
author={Yihe Dong and Jean-Baptiste Cordonnier and Andreas Loukas},
journal={arXiv: Attention and Vision},
year={2023}
}
Note: Given the scope of the information, I've categorized the article under the journal arXiv: Attention and Vision based on the topic. Adjust the "journal" field as necessary to reflect the correct source or preprint server (e.g., journal={arXiv preprint arXiv:xxxx.xxxx}) if this is more appropriate or if the document has a specific identifier not provided.
[82] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme Ruiz, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjoerd Van Steenkiste, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Collier, Alexey A. Gritsenko, Vighnesh Birodkar, Cristina Nader Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetic, Dustin Tran, Thomas Kipf, Mario Lucic, Xiaohua Zhai, Daniel Keysers, Jeremiah J. Harmsen, and Neil Houlsby. Scaling vision transformers to 22 billion parameters. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 7480-7512. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr. press/v202/dehghani23a.html. Please convert into a bibtex citation.
AI:
@inproceedings{dehghani2023scaling,
title={Scaling vision transformers to 22 billion parameters},
author={Dehghani, Mostafa and Djolonga, Josip and Mustafa, Basil and Padlewski, Piotr and Heek, Jonathan and Gilmer, Justin and Steiner, Andreas Peter and Caron, Mathilde and Geirhos, Robert and Alabdulmohsin, Ibrahim and Jenatton, Rodolphe and Beyer, Lucas and Tschannen, Michael and Arnab, Anurag and Wang, Xiao and Riquelme Ruiz, Carlos and Minderer, Matthias and Puigcerver, Joan and Evci, Utku and Kumar, Manoj and Van Steenkiste, Sjoerd and Elsayed, Gamaleldin Fathy and Mahendran, Aravindh and Yu, Fisher and Oliver, Avital and Huot, Fantine and Bastings, Jasmijn and Collier, Mark and Gritsenko, Alexey A. and Birodkar, Vighnesh and Vasconcelos, Cristina Nader and Tay, Yi and Mensink, Thomas and Kolesnikov, Alexander and Pavetic, Filip and Tran, Dustin and Kipf, Thomas and Lucic, Mario and Zhai, Xiaohua and Keysers, Daniel and Harmsen, Jeremiah J. and Houlsby, Neil},
booktitle={Proceedings of the 40th International Conference on Machine Learning},
volume={202},
pages={7480--7512},
year={2023},
editor={Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan},
organization={PMLR},
url={https://proceedings.mlr.press/v202/dehghani23a.html},
conference={ICML 2023}
}
[83] Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie E Everett, Alexander A Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha SohlDickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, and Simon Kornblith. Small-scale proxies for largescale transformer training instabilities. In The Twelfth Please convert into a bibtex citation.
AI:
@inproceedings{wortsman2023small,
title={Small-scale proxies for large-scale transformer training instabilities},
author={Wortsman, Mitchell and Liu, Peter J and Xiao, Lechao and Everett, Katie E and Alemi, Alexander A and Adlam, Ben and Co-Reyes, John D and Gur, Izzeddin and Kumar, Abhishek and Novak, Roman and Pennington, Jeffrey and Sohl-Dickstein, Jascha and Xu, Kelvin and Lee, Jaehoon and Gilmer, Justin and Kornblith, Simon},
booktitle={The Twelfth},
year={2023}
}
International Conference on Learning Representations, 2024. URL https: / openreview. net/ forum?id=d8w0pmvXbZ.
[84] Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zeroshot hyperparameter transfer. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 17084-17097. Curran Associates, Inc., 2021. URL https://proceedings.neurips. cc/paper_files/paper/2021/file/ 8df7c2e3c3c3be098ef7b382bd2c37ba-Paper. $\mathrm{pdf}$. Please convert into a bibtex citation.
AI:
@inproceedings{Yang2021TuningLN,
title={Tuning Large Neural Networks via Zero-shot Hyperparameter Transfer},
author={Ge Yang and Edward Hu and Igor Babuschkin and Szymon Sidor and Xiaodong Liu and David Farhi and Nick Ryder and Jakub Pachocki and Weizhu Chen and Jianfeng Gao},
booktitle={Advances in Neural Information Processing Systems},
volume={34},
pages={17084--17097},
year={2021},
editor={M. Ranzato and A. Beygelzimer and Y. Dauphin and P. S. Liang and J. Wortman Vaughan},
publisher={Curran Associates, Inc.},
url={https://proceedings.neurips.cc/paper_files/paper/2021/file/8df7c2e3c3c3be098ef7b382bd2c37ba-Paper.pdf}
}
[85] Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Tensor programs VI: Feature learning in infinite depth neural networks. In The Twelfth International Conference on Learning Representations, 2024. URL https : / /openreview. net/ forum?id=17pVDnpwwl. Please convert into a bibtex citation.
AI: Given the provided details, a BibTeX citation for this reference could be constructed as follows:
@inproceedings{yang2024tensor,
title={Tensor Programs VI: Feature Learning in Infinite Depth Neural Networks},
author={Yang, Greg and Yu, Dingli and Zhu, Chen and Hayou, Soufiane},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=17pVDnpwwl}
}
This BibTeX entry includes all the essential fields such as title, authors, booktitle, year, and URL, formatted according to typical citation standards. Make sure you integrate this correctly into your BibTeX database, and adjust it accordingly if your document or bibliography management tool requires any specific formatting or field additions.
[86] Jürgen Haas, Alessandro Barbato, Dario Behringer, Gabriel Studer, Steven Roth, Martino Bertoni, Khaled Mostaguir, Rafal Gumienny, and Torsten Schwede. Continuous Automated Model EvaluatiOn (CAMEO) complementing the critical assessment of structure prediction in CASP12. Proteins: Structure, Function and Bioinformatics, 86(Suppl 1):387-398, March 2018. ISSN 10970134. doi: 10.1002/prot.25431. Publisher: John Wiley and Sons Inc. Please convert into a bibtex citation.
AI: Certainly, here is how you could format this reference as a BibTeX entry:
@article{Haas2018CAMEO,
author = {J{\"u}rgen Haas and Alessandro Barbato and Dario Behringer
and Gabriel Studer and Steven Roth and Martino Bertoni
and Khaled Mostaguir and Rafal Gumienny and Torsten Schwede},
title = {{Continuous Automated Model EvaluatiOn (CAMEO) complementing
the critical assessment of structure prediction in CASP12}},
journal = {Proteins: Structure, Function, and Bioinformatics},
volume = {86},
number = {Suppl 1},
pages = {387--398},
year = {2018},
month = mar,
issn = {10970134},
doi = {10.1002/prot.25431},
publisher = {John Wiley and Sons Inc}
}
[87] Andriy Kryshtafovych, Torsten Schwede, Maya Topf, Krzysztof Fidelis, and John Moult. Critical assessment of methods of protein structure prediction (CASP)—Round XIV. Proteins: Structure, Function, and Bioinformatics, 89(12):1607-1617, 2021. ISSN 1097-0134. doi: 10.1002/prot.26237. URL https://onlinelibrary.wiley.com/ doi/abs/10.1002/prot.26237. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/prot.26237. Please convert into a bibtex citation.
AI:
@article{Kryshtafovych2021,
author = {Kryshtafovych, Andriy and Schwede, Torsten and Topf, Maya and Fidelis, Krzysztof and Moult, John},
title = {Critical assessment of methods of protein structure prediction (CASP)—Round XIV},
journal = {Proteins: Structure, Function, and Bioinformatics},
volume = {89},
number = {12},
pages = {1607-1617},
year = {2021},
doi = {10.1002/prot.26237},
ISSN = {1097-0134},
url = {https://onlinelibrary.wiley.com/doi/abs/10.1002/prot.26237},
eprint = {https://onlinelibrary.wiley.com/doi/pdf/10.1002/prot.26237}
}
[88] Andriy Kryshtafovych, Maciej Antczak, Marta Szachniuk, Tomasz Zok, Rachael C. Kretsch, Ramya Rangan, Phillip Pham, Rhiju Das, Xavier Robin, Gabriel Studer, Janani Durairaj, Jerome Eberhardt, Aaron Sweeney, Maya Topf, Torsten Schwede, Krzysztof Fidelis, and John Moult. New prediction categories in CASP15. Proteins, 91(12):1550-1557, December 2023. ISSN 0887-3585. doi: 10.1002/prot. 26515. URL https://www.ncbi.nlm.nih. gov/pmc/articles/PMC10713864/. Please convert into a bibtex citation.
@article{Kryshtafovych2023NewPrediction, title={New prediction categories in CASR15}, author={Kryshtafovych, Andriy and Antczak, Maciej and Szachniuk, Marta and Zok, Tomasz and Kretsch, Rachael C. and Rangan, Ramya and Pham, Phillip and Das, Rhiju and Robin, Xavier and Studer, Gabriel and Durairaj, Janani and Eberhardt, Jerome and Sweeney, Aaron and Topf, Maya and Schwede, Torsten and Fidelis, Krzysztof and Moult, John}, journal={Proteins}, volume={91}, number={12}, pages={1550-1557}, year={2023}, month={December}, publisher={Wiley}, doi={10.1002/prot.26515}, issn={0887-3585}, url={https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10713864/} }
[89] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models, October 2021. URL http://arxiv.org/abs/2106.09685. arXiv:2106.09685 [cs]
Please convert into a bibtex citation.
[AI]: Sure, here is the BibTeX citation for the document you've provided:
bibtex @misc{hu2021lora, title={LoRA: Low-Rank Adaptation of Large Language Models}, author={Edward J. Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen}, year={2021}, eprint={2106.09685}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={http://arxiv.org/abs/2106.09685}, month={oct} }
This citation includes the title, authors, year of publication, eprint number, archive prefix, primary classification, and URL to the document on arXiv. You can use this entry in a BibTeX database for referencing in academic work. Please make sure that you are using the right formatting guidelines for your specific purpose, as different venues might require slight variations in the way BibTeX entries are structured.
[90] Leland McInnes, John Healy, and James Melville. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction, September 2020. URLhttp://arxiv.org/abs/1802.03426. arXiv:1802.03426 [cs, stat].
Please convert into a bibtex citation.
[AI]:
bibtex @article{mcinnes2020umap, title={UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction}, author={McInnes, Leland and Healy, John and Melville, James}, journal={arXiv preprint arXiv:1802.03426}, year={2020}, month={Sep}, url={http://arxiv.org/abs/1802.03426}, archivePrefix={arXiv}, eprint={1802.03426}, primaryClass={cs, stat} }
[91] Brian Hie, Salvatore Candido, Zeming Lin, Ori Kabeli, Roshan Rao, Nikita Smetanin, Tom Sercu, and Alexander Rives. A high-level programming language for generative protein design. bioRxiv, pages 2022-12, 2022.
Please convert into a bibtex citation.
[AI]:
bibtex @article{hie2022high, title={A high-level programming language for generative protein design}, author={Hie, Brian and Candido, Salvatore and Lin, Zeming and Kabeli, Ori and Rao, Roshan and Smetanin, Nikita and Sercu, Tom and Rives, Alexander}, journal={bioRxiv}, pages={2022-12}, year={2022} }
[92] Nicolas Hulo, Amos Bairoch, Virginie Bulliard, Lorenzo Cerutti, Edouard De Castro, Petra S. Langendijk-Genevaux, Marco Pagni, and Christian J. A. Sigrist. The PROSITE database. Nucleic Acids Research, 34(Database issue):D227-230, January 2006. ISSN 1362-4962. doi: 10.1093/nar/gkj063.
Please convert into a bibtex citation.
[AI]:
bibtex @article{Hulo2006, title={The PROSITE database}, author={Hulo, Nicolas and Bairoch, Amos and Bulliard, Virginie and Cerutti, Lorenzo and De Castro, Edouard and Langendijk-Genevaux, Petra S. and Pagni, Marco and Sigrist, Christian J. A.}, journal={Nucleic Acids Research}, volume={34}, number={Database issue}, pages={D227-230}, year={2006}, month={Jan}, publisher={Oxford University Press}, issn={1362-4962}, doi={10.1093/nar/gkj063} }
[93] Chengxin Zhang, Xi Zhang, Peter L Freddolino, and Yang Zhang. BioLiP2: an updated structure database for biologically relevant ligand-protein interactions. Nucleic Acids Research, 52(D1):D404D412, 07 2023. ISSN 0305-1048. doi: 10.1093/nar/ gkad630. URL https://doi.org/10.1093/ nar/gkad630.
Please convert into a bibtex citation.
[AI]:
bibtex @article{zhang2023biolip2, title={BioLiP2: an updated structure database for biologically relevant ligand-protein interactions}, author={Zhang, Chengxin and Zhang, Xi and Freddolino, Peter L and Zhang, Yang}, journal={Nucleic Acids Research}, volume={52}, number={D1}, pages={D404--D412}, year={2023}, publisher={Oxford University Press}, doi={10.1093/nar/gkad630}, issn={0305-1048}, url={https://doi.org/10.1093/nar/gkad630}, month={07} }
[94] Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 8946-8970. PMLR, June 2022. URL https://proceedings.mlr. press/v162/hsu22a.html. ISSN: 2640-3498.
Please convert into a bibtex citation.
[AI]:
bibtex @inproceedings{hsu2022learning, title={Learning inverse folding from millions of predicted structures}, author={Hsu, Chloe and Verkuil, Robert and Liu, Jason and Lin, Zeming and Hie, Brian and Sercu, Tom and Lerer, Adam and Rives, Alexander}, booktitle={Proceedings of the 39th International Conference on Machine Learning}, volume={162}, pages={8946--8970}, year={2022}, month={June}, editor={Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, series={Proceedings of Machine Learning Research}, publisher={PMLR}, url={https://proceedings.mlr.press/v162/hsu22a.html}, issn={2640-3498} }
[95] Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A General Theoretical Paradigm to Understand Learning from Human Preferences, November 2023. URL http: / / arxiv. org/abs/2310.12036. arXiv:2310.12036 [cs, stat].
Please convert into a bibtex citation.
[AI]:
bibtex @article{Azar2023AGeneral, title={A General Theoretical Paradigm to Understand Learning from Human Preferences}, author={Mohammad Gheshlaghi Azar and Mark Rowland and Bilal Piot and Daniel Guo and Daniele Calandriello and Michal Valko and Rémi Munos}, journal={arXiv preprint arXiv:2310.12036}, year={2023}, month={Nov}, url={http://arxiv.org/abs/2310.12036}, archivePrefix={arXiv}, eprint={2310.12036}, primaryClass={cs, stat} }
[96] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: Model Alignment as Prospect Theoretic Optimization, June 2024.
Please convert into a bibtex citation.
[AI]: Since my last update was in September 2023, and I can't predict future publications, I'll create a fictional BibTeX citation based on your input as if the information provided were accurate. Note that for actual academic writing, you'll want to verify the specifics such as the conference or journal details, volume, and pages which are not provided in your question. I'll include a placeholder for these details.
bibtex @inproceedings{ethayarajh2024kto, title={KTO: Model Alignment as Prospect Theoretic Optimization}, author={Ethayarajh, Kawin and Xu, Winnie and Muennighoff, Niklas and Jurafsky, Dan and Kiela, Douwe}, booktitle={Proceedings of the [Conference Name]}, volume={XX}, number={}, pages={XX--XX}, year={2024}, organization={[Organization]}, month={Jun}, publisher={[Publisher]}, address={[Location]}, note={}, doi={}, url={}, }
Make sure to replace the placeholders (`[Conference Name]`, `[Organization]`, `[Publisher]`, `[Location]`, `XX`, etc.) with the actual values relevant to the publication.
URL http://arxiv.org/abs/2402.01306. arXiv:2402.01306 [cs].
[97] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
Please convert into a bibtex citation.
[AI]:
bibtex @inproceedings{gao2023scaling, title={Scaling laws for reward model overoptimization}, author={Gao, Leo and Schulman, John and Hilton, Jacob}, booktitle={Proceedings of the 40th International Conference on Machine Learning}, series={ICML'23}, year={2023}, organization={JMLR.org} }
[98] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.
Please convert into a bibtex citation.
[AI]: ```
@article{chen2021evaluating,
title={Evaluating large language models trained on code},
author={Chen, Mark and Tworek, Jerry and Jun, Heewoo and Yuan, Qiming and de Oliveira Pinto, Henrique Ponde and Kaplan, Jared and Edwards, Harri and Burda, Yuri and Joseph, Nicholas and Brockman, Greg and Ray, Alex and Puri, Raul and Krueger, Gretchen and Petrov, Michael and Khlaaf, Heidy and Sastry, Girish and Mishkin, Pamela and Chan, Brooke and Gray, Scott and Ryder, Nick and Pavlov, Mikhail and Power, Alethea and Kaiser, Lukasz and Bavarian, Mohammad and Winter, Clemens and Tillet, Philippe and Such, Felipe Petroski and Cummings, Dave and Plappert, Matthias and Chantzis, Fotios and Barnes, Elizabeth and Herbert-Voss, Ariel and Guss, William Hebgen and Nichol, Alex and Paino, Alex and Tezak, Nikolas and Tang, Jie and Babuschkin, Igor and Balaji, Suchir and Jain, Shantanu and Saunders, William and Hesse, Christopher and Carr, Andrew N. and Leike, Jan and Achiam, Josh and Misra, Vedant and Morikawa, Evan and Radford, Alec and Knight, Matthew and Brundage, Miles and Murati, Mira and Mayer, Katie and Welinder, Peter and McGrew, Bob and Amodei, Dario and McCandlish, Sam and Sutskever, Ilya and Zaremba, Wojciech},
journal={Your Journal Name},
volume={Volume Number},
number={Issue Number},
pages={Page Range},
year={2021},
publisher={Publisher}
}
Please adjust the journal, volume, number, pages, and publisher field with the specific details of where the article was published. The placeholder text "Your Journal Name", "Volume Number", "Issue Number", "Page Range", and "Publisher" should be replaced with the actual data relevant to this publication.
[99] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. Please convert into a bibtex citation.
AI:
@article{ho2022classifier,
title={Classifier-free diffusion guidance},
author={Ho, Jonathan and Salimans, Tim},
journal={arXiv preprint arXiv:2207.12598},
year={2022}
}
[100] W. Kabsch. A solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A, 32(5):922-923, 1976. doi: https://doi.org/10.1107/S0567739476001873. URL https://onlinelibrary.wiley.com/ doi/abs/10.1107/S0567739476001873. Please convert into a bibtex citation.
AI:
@article{Kabsch1976,
title={A solution for the best rotation to relate two sets of vectors},
author={Kabsch, W.},
journal={Acta Crystallographica Section A},
volume={32},
number={5},
pages={922--923},
year={1976},
publisher={International Union of Crystallography},
doi={10.1107/S0567739476001873},
url={https://onlinelibrary.wiley.com/doi/abs/10.1107/S0567739476001873}
}
[101] Sophia M. Hartley, Kelly A. Tiernan, Gjina Ahmetaj, Adriana Cretu, Yan Zhuang, and Marc Zimmer. AlphaFold2 and RoseTTAFold predict posttranslational modifications. Chromophore formation in GFP-like proteins. PLOS ONE, 17 (6):e0267560, June 2022. ISSN 1932-6203. doi: 10.1371/journal.pone.0267560. URL https:// journals.plos.org/plosone/article? id=10.1371/ journal.pone. 0267560 . Please convert into a bibtex citation.
AI: Certainly! Here's how the citation for the mentioned article can be formatted in BibTeX:
@article{hartley2022alphafold2,
title={AlphaFold2 and RoseTTAFold predict posttranslational modifications. Chromophore formation in GFP-like proteins},
author={Hartley, Sophia M and Tiernan, Kelly A and Ahmetaj, Gjina and Cretu, Adriana and Zhuang, Yan and Zimmer, Marc},
journal={PLOS ONE},
volume={17},
number={6},
pages={e0267560},
year={2022},
month={June},
publisher={Public Library of Science},
doi={10.1371/journal.pone.0267560},
url={https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0267560},
issn={1932-6203}
}
Please make sure to check for any specific requirements or variations your institution or publisher may have for BibTeX entries.
Publisher: Public Library of Science.
[102] Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. Masked language model scoring. arXiv:1910.14659, 2019. Please convert into a bibtex citation.
AI: Here is how you could format the citation in BibTeX:
@article{salazar2019masked,
title={Masked language model scoring},
author={Salazar, Julian and Liang, Davis and Nguyen, Toan Q and Kirchhoff, Katrin},
journal={arXiv preprint arXiv:1910.14659},
year={2019}
}
This is a common format for citing preprints from arXiv in a BibTeX database. It includes the title of the paper, the authors, the preprint identifier (arXiv ID), and the year of publication.
[103] L.G. Somermeyer. Orthologous gfp fitness peaks. https://archive. softwareheritage.org/swh:1:cnt:a4c63cdf2f4524c8d5c813a1972a5ac649266e2b, 2022. Please convert into a bibtex citation.
AI: Here's how you could format this citation in BibTeX:
@misc{Somermeyer2022,
author = {Somermeyer, L.G.},
title = {Orthologous gfp fitness peaks},
year = {2022},
url = {https://archive.softwareheritage.org/swh:1:cnt:a4c63cdf2f4524c8d5c813a1972a5ac649266e2b},
note = {Accessed: [your access date here]}
}
Please remember to replace [your access date here] with the actual date you accessed this resource. The @misc tag is used here because this citation doesn't fit neatly into more specific categories such as @article or @book.
[104] Kazutaka Katoh and Daron M Standley. Mafft multiple sequence alignment software version 7: improvements in performance and usability. Molecular biology and evolution, 30(4):772-780, 2013. Please convert into a bibtex citation.
AI: Here is your request in BibTeX format:
@article{Katoh2013,
title={MAFFT multiple sequence alignment software version 7: improvements in performance and usability},
author={Katoh, Kazutaka and Standley, Daron M},
journal={Molecular biology and evolution},
volume={30},
number={4},
pages={772-780},
year={2013},
publisher={Oxford University Press}
}
[105] Talley J. Lambert. FPbase: a communityeditable fluorescent protein database. Nature Methods, 16(4):277-278, April 2019. ISSN 1548-7105. doi: 10.1038/s41592-019-0352-8. URL https://www.nature.com/articles/ s41592-019-0352-8. Publisher: Nature Publishing Group. Please convert into a bibtex citation.
AI:
@article{Lambert2019,
author = {Lambert, Talley J.},
title = {{FPbase: a communityeditable fluorescent protein database}},
journal = {Nature Methods},
volume = {16},
number = {4},
pages = {277-278},
year = {2019},
doi = {10.1038/s41592-019-0352-8},
url = {https://www.nature.com/articles/s41592-019-0352-8},
publisher = {Nature Publishing Group},
issn = {1548-7105}
}
[106] Skipper Seabold and Josef Perktold. statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference, 2010. Please convert into a bibtex citation.
AI:
@inproceedings{seabold2010statsmodels,
title={statsmodels: Econometric and statistical modeling with python},
author={Seabold, Skipper and Perktold, Josef},
booktitle={9th Python in Science Conference},
year={2010}
}
[107] Responsible AI x Biodesign Responsible AI x Biodesign. Responsible AI x biodesign. https: //responsiblebiodesign.ai/, 2024. Accessed: 2024-6-20. Please convert into a bibtex citation.
AI:
@misc{responsible_ai_biodesign_2024,
title = {Responsible AI x Biodesign},
url = {https://responsiblebiodesign.ai/},
year = {2024},
note = {Accessed: 2024-6-20}
}
[108] Center for Disease Control. Select agents and toxins list. https://www.selectagents.gov/ sat/list.htm, May 2024. Accessed: 2024-5-24. Please convert into a bibtex citation.
AI: Certainly! Below is how you would format the citation as a BibTeX entry. Since you provided information with a future date that I cannot predict, I will proceed with the information as you've given, but please keep in mind that my response is based on data and structures known up to 2023.
@misc{CDC2024,
author = "{Center for Disease Control}",
title = "{Select Agents and Toxins List}",
year = 2024,
month = may,
howpublished = "\url{https://www.selectagents.gov/sat/list.htm}",
note = "Accessed: 2024-5-24"
}
This BibTeX entry includes the basic elements: type of document (misc, since it's a web source), author (though it's an organization, it's listed as the author here), title of the document, year and month of publication, the howpublished field to indicate a URL, and a note field for the access date, which is a common requirement for citing web resources.
Please adjust the entry as necessary to fit the requirements of the specific citation style you are supposed to follow or any additional information you may need to include.
[109] Department of Human Health Services. Screening framework guidance for providers and users of synthetic nucleic acids. Technical report, 2023. URL https://aspr.hhs.gov/legal/synna/ Documents/SynNA-Guidance-2023.pdf. Please convert into a bibtex citation.
AI: Here's a BibTeX citation for the document you've mentioned:
@techreport{DHHS2023,
author = "{{Department of Human Health Services}}",
title = "{Screening framework guidance for providers and users of synthetic nucleic acids}",
year = 2023,
type = "Technical Report",
institution = "Department of Human Health Services",
url = "https://aspr.hhs.gov/legal/synna/Documents/SynNA-Guidance-2023.pdf"
}
Just a little note: I've put "Department of Human Health Services" in double braces {{ }} in the author field to prevent BibTeX from assuming that "Department" is a first name and "of" is a middle name, etc. This keeps the entire institution name intact as a single unit. Also, please note that the provided citation assumes "Department of Human Health Services" is the correct name; however, it typically might be "Department of Health and Human Services" (HHS) in certain contexts, so please adjust it as necessary for accuracy.
[110] Pascal Notin, Aaron W Kollasch, Daniel Ritter, Lood van Niekerk, Steffanie Paul, Hansen Spinner, Nathan Rollins, Ada Shaw, Ruben Weitzman, Jonathan Frazer, Mafalda Dias, Dinko Franceschi, Rose Orenbuch, Yarin Gal, and Debora S Marks. ProteinGym: Large-scale benchmarks for protein design and fitness prediction. bioRxiv, page 2023.12.07.570727, December 2023. URL https://www.biorxiv.org/content/10. $1101 / 2023.12 .07 .570727 v 1$.
[111] Thomas A Hopf, John B Ingraham, Frank J Poelwijk, Charlotta PI Schärfe, Michael Springer, Chris Sander, and Debora S Marks. Mutation effects predicted from sequence co-variation. Nature biotechnology, 35(2):128, February 2017. ISSN 15461696. doi: 10.1038/nbt.3769. URL http://www. nature. com/articles/nbt.3769. Publisher: Nature Publishing Group. Please convert into a bibtex citation.
AI: Certainly! Here's a BibTeX citation for the article you provided:
@article{Hopf2017MutationEffects,
title={Mutation effects predicted from sequence co-variation},
author={Hopf, Thomas A and Ingraham, John B and Poelwijk, Frank J and Sch{\"a}rfe, Charlotta PI and Springer, Michael and Sander, Chris and Marks, Debora S},
journal={Nature Biotechnology},
volume={35},
number={2},
pages={128},
year={2017},
month={February},
publisher={Nature Publishing Group},
issn={15461696},
doi={10.1038/nbt.3769},
url={http://www.nature.com/articles/nbt.3769}
}
Note: Ensure to check for any specific formatting requirements your citation manager or citation style may have and adjust accordingly.
ME: Please list all the authors
AI: Certainly! Here is a list of all the authors from the provided citations. Please note, for citations with numerous authors, I've accurately transcribed the author names as provided, including where "et al." may have been used in the initial citation requests you provided:
(Note: For citations where "et al." was used or indicated, only the explicitly named authors are listed here, as per your request. The "et al." indicates additional authors not listed explicitly.)
ME: what are the most cited authors?
AI: Based on the citations provided across all your requests, it's not immediately clear which authors are the most cited without analyzing the entire list for repeat occurrences, which could require a more comprehensive and analytical method beyond manual inspection, especially considering the vast diversity of the fields involved (ranging from bioinformatics to computer science).
However, a few names stand out due to their appearance in multiple distinct citations:
For a precise and comprehensive determination of "the most cited authors" based on your list, one would ideally employ bibliometric tools or databases that track citation counts and frequencies, such as Google Scholar, Scopus, or Web of Science. These platforms can provide citation metrics for authors, including total citations, h-index, and i10-index, which offer a more quantifiable measure of an author’s impact and visibility within the scientific community.
It is important to note that citation frequency can be highly field-dependent, with certain fields having generally higher citation rates due to their nature or the number of researchers working within them.
ME: which authors are mentioned in the most papers here
AI: Based on the provided list and without a distinguishing count for each mention due to the format of the input, identifying authors mentioned in multiple unique citations provides an approximation of those with contributions across several papers. From the citations provided and examining for repeat mentions, a few authors stand out:
These authors are associated with cutting-edge research in their respective fields, contributing to transformative papers and projects that span across computational biology, artificial intelligence, genomics, and structural bioinformatics. Their repeat mentions across the citations signify their leadership and influence in these interdisciplinary areas of study.
Since the citations cover a broad spectrum of disciplines and the compilation is inherently selective, it’s crucial to recognize that "most mentioned" within this list may not fully encapsulate an author's influence or the broader landscape of citations in the scientific community. Tools like Google Scholar, Scopus, or Web of Science are best suited for comprehensive bibliometric analyses to determine author prominence and citation impact systematically.
ME: what journals are represented?
AI: The citations you provided span a variety of journals, reflecting the diversity of research fields, particularly within biological sciences, computer science, and interdisciplinary areas. Here's a summary of journals mentioned in your list:
These journals/conferences are well-regarded in their respective fields, ranging from bioinformatics and protein research to machine learning and artificial intelligence. They reflect the interdisciplinary nature of modern scientific inquiries, where computational techniques are increasingly employed to solve complex biological problems. Nature Biotechnology and Nucleic Acids Research are particularly notable for their focus on the application of technology and bioinformatics in biology and medicine. IEEE Transactions on Pattern Analysis and Machine Intelligence emphasizes computer science and artificial intelligence research.
Each of these journals and conference proceedings plays a crucial role in disseminating new knowledge and fostering innovation across disciplines. The mentioned titles are associated with high-quality research and peer review standards, contributing significantly to their respective scientific communities.
Thomas A Hopf John B Ingraham Frank J Poelwijk Charlotta PI Schärfe Michael Springer Chris Sander Debora S Marks Pascal Notin Aaron W Kollasch Daniel Ritter Lood van Niekerk Steffanie Paul Hansen Spinner Nathan Rollins Ada Shaw Ruben Weitzman Jonathan Frazer Mafalda Dias Dinko Franceschi Rose Orenbuch Yarin Gal Edward J Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen Leland McInnes John Healy James Melville Brian Hie Salvatore Candido Zeming Lin Ori Kabeli Roshan Rao Nikita Smetanin Tom Sercu Adam Lerer Kazutaka Katoh Daron M Standley Talley J Lambert Skipper Seabold Josef Perktold Andriy Kryshtafovych Maciej Antczak Marta Szachniuk Tomasz Zok Rachael C Kretsch Ramya Rangan Phillip Pham Rhiju Das Xavier Robin Gabriel Studer Janani Durairaj Jerome Eberhardt Aaron Sweeney Maya Topf Torsten Schwede Krzysztof Fidelis John Moult Chengxin Zhang Xi Zhang Peter L Freddolino Yang Zhang Chloe Hsu Robert Verkuil Jason Liu Brian Hie Tom Sercu Adam Lerer Alexander Rives Mohammad Gheshlaghi Azar Mark Rowland Bilal Piot Daniel Guo Daniele Calandriello Michal Valko Rémi Munos Kawin Ethayarajh Winnie Xu Niklas Muennighoff Dan Jurafsky Douwe Kiela Jürgen Haas Alessandro Barbato Dario Behringer Gabriel Studer Steven Roth Martino Bertoni Khaled Mostaguir Rafal Gumienny Torsten Schwede Nicolas Hulo Amos Bairoch Virginie Bulliard Lorenzo Cerutti Edouard De Castro Petra S Langendijk-Genevaux Marco Pagni Christian J A Sigrist Julian Salazar Davis Liang Toan Q Nguyen Katrin Kirchhoff Leo Gao John Schulman Jacob Hilton Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever Wojciech Zaremba Julian B Kinney
A Materials and Methods ….. 21 A. 1 Architecture ….. 21 A.1.1 Notation ….. 21 A.1.2 Overview ….. 21 A.1.3 Tokenization ….. 21 A.1.4 ESM3 Inputs and Forward Pass ….. 21 A.1.5 Transformer ….. 23 A.1.6 Geometric Attention ….. 24 A.1.7 Structure Tokenizer ….. 26 A.1.8 Function Tokenization ….. 31 A.1.9 Other Tracks ….. 36 A.1.10 ESM3 Inference ….. 37 A. 2 Training ESM3 ….. 37 A.2.1 Pre-training Data ….. 37 A.2.2 Pre-training Tasks ….. 39 A.2.3 Training Details ….. 41 A. 3 Model evaluations ….. 42 A.3.1 Models ….. 42 A.3.2 Data ….. 42 A.3.3 Representation Learning ….. 42 A.3.4 Structure Prediction ….. 43 A.3.5 Conditional Likelihood ….. 43 A.3.6 Unconditional Generation ….. 43 A.3.7 Prompt-following Evaluations ….. 46 A.3.8 Steerable Design ….. 49 A.3.9 Composing Prompts ….. 49 A.3.10 Multimodal Editing Examples ….. 51 A. 4 Alignment ….. 53 A.4.1 Algorithm ….. 53 A.4.2 Preference Tuning Intuition ….. 54 A.4.3 Evaluation Metrics ….. 54 A.4.4 Training Dataset ….. 55 A.4.5 Evaluation Dataset: Atomic Coordination ….. 55 A.4.6 Supervised Finetuning ….. 55 A.4.7 Training Hyperparameters ….. 55 A. 5 GFP ….. 55 A.5.1 Generation and Selection ….. 59 A.5.2 Experimental Methods and Data Analysis ….. 61 A.5.3 Sequence searches and comparisons ….. 62 A.5.4 Phylogenetic Analysis ….. 63 A. 6 Open model ….. 64 A.6.1 ESM3-open Mitigations ….. 64 A.6.2 ESM3-open Evaluations ….. 67
In the following, we use $L$ to denote the sequence length, $d$ for the embedding dimension, ${a . . b}$ to denote the inclusive set of integers from $a$ to $b$, and $[a, b]$ an interval of real numbers. $S E(3)$ is the special Euclidean group, which we use to denote frames (Appendix A.1.6.1).
ESM3 is all-to-all generative model that both conditions on and generates a variety of different tracks. As input, ESM3 is conditioned on various tracks as described in Appendix A.1.5.1, and as output, ESM3 generates predictions detailed in Appendix A.1.5.2.
The generative pipeline is as follows.
Tokenization First, raw inputs are tokenized as described in Appendix A.1.3. Structural inputs are tokenized via a VQ-VAE (Appendix A.1.7). Function keywords are tokenized by quantizing the TF-IDF transform of functional keywords with locality sensitive hashing (LSH), detailed in Appendix A.1.8.
Transformer Trunk A standard Transformer $(57,58)$ architecture processes the post-tokenized inputs. Geometric Attention (Algorithm 6 and Fig. S2) directly processes structural coordinates as input. Model outputs are logits over token space, and can be sampled to obtain outputs described in Appendix A.1.5.2. The overall architecture is diagrammed in Fig. S1.
Decoder Most tracks can be naively decoded into tokens detailed in Appendix A.1.3. Structure tokens must be decoded with a model - we use a $700 \mathrm{M}$ parameter transformer model to do this, trained post-hoc (Appendix A.1.7.2). The decoder uses sequence tokens and structure tokens to directly predict coordinates, pTM, and pLDDT (59). Function tokens are decoded using a small 3-layer transformer, trained post-hoc to invert the LSH quantization procedure (Appendix A.1.8.2.1).
During tokenization, special beginning-of-sequence (BOS) and end-of-sequence (EOS) tokens are prepended and appended to mark the real start of sequences. When sequences are cropped due to length, the BOS and EOS tokens are cropped out to indicate protein fragments. In all cases, one token per track is used for each amino acid.
Sequence Protein sequences are tokenized as the 20 canon- ical amino acids, plus BOS, EOS, mask, pad, unknown. We keep four non-standard amino acids as in Lin et al. (5), B - Asparagine, U - selenocysteine, Z - glutamic acid, and O - ornithine. This totals to 29 tokens.
Structure Structure tokenization is described in Appendix A.1.7.1. ESM3 uses a codebook size of 4096 with 4 special tokens - EOS, BOS, mask, and pad.
Secondary Structure Secondary structure is taken to be the canonical 8-class tokens (60), with unknown and mask, for a total of 10 tokens. The mask token is forced to be the 0 -vector during embedding.
SASA The continuous values representing SASA are tokenized by discretization into a fixed set of 16 bins. SASA bin boundaries were chosen by computing SASA on 100 random structures and ensuring an equal number of residues belong in each bin. Unknown and mask are used for a total of 18 tokens. The mask token is forced to be the 0 -vector during embedding.
Function annotations We tokenize function annotations as bags of keywords, described in Appendix A.1.8. Keywords are quantized using LSH into 8 tokens per residue, each of which can be one of 255 tokens. There are three special tokens, empty set, no-annotation, and mask. Again, the mask token is forced to be the 0 vector during embedding.
Residue annotations InterPro annotations are tokenized as a multi-hot feature vector (1478 dimensions) over possible InterPro labels (38). Input annotations are limited to a maximum of 16 . When annotations are not present, we enforce that the 0 -vector is added.
As mentioned above, ESM3 can take several tracks, all of which are optionally disabled via masking. In the following, we concisely denote the inputs to ESM3 as
$$ \mathbf{x}{\text {inputs }}=\left{\begin{array}{l} x{\text {structure }} \in{0 . .4099}^{L}, x{\mathrm{ss} 8} \in{0 . .10}^{L} \ x{\text {sasa }} \in{0 . .18}^{L}, x{\mathrm{func}} \in{0 . .258}^{L \times 8} \ x{\mathrm{res}} \in{0,1}^{L \times 1478}, x{\mathrm{res}} \in{0,1}^{L \times 1478} \ x{\text {plddt }} \in[0,1]^{L}, x_{\text {avgplddt }} \in[0,1] \end{array}\right. $$
We now present the high level algorithm for a forward pass of ESM3:
Figure S1. The ESM3 architecture. ESM3 is a masked language model that reasons over protein sequence, structure, and function, each of which are represented as token tracks at the input and output. Tokens are embedded and summed at the input to a transformer stack. The first block (expanded on the right) contains an additional geometric attention layer for processing atomic coordinate inputs. During training, random masks are sampled and applied to each track. Masked token positions are predicted at the output.
Algorithm 1 esm3_forward
Input: $\mathbf{x}_{\text {inputs }}$
1: $z_{\text {embed }}^{(0)}=$ encode_inputs $\left(\mathbf{x}_{\text {inputs }}\right) \quad \triangleright \mathbb{R}^{L \times d}$
for $\ell \in\left\{1 . . n_{\text {layers }}\right\}$ do
$z_{\text {embed }}^{(\ell)}=$ transformer__block $\left(z_{\text {embed }}^{(\ell-1)}\right)$
end for
for track in desired output tracks do
$z_{\text {track }}=$ regression_head $\left(z_{\text {embed }}^{\left(n_{\text {layers }}\right)}\right)$
end for
return Track specific logits $z_{\text {track }} \in \mathbb{R}^{L \times c_{\text {track }}}$
In the next few sections, we detail each component.
Our network is based on the transformer architecture (57), incorporating several subsequent improvements: We use Pre-LN instead of Post-LN (58), rotary embeddings (61) instead of absolute positional embeddings, and we replace ReLU non-linearity with SwiGLU (62). The hidden dimension is set to approximately $\frac{8}{3} d$, rounded to the nearest multiple of 256 for training efficiency. No biases are used in linear layers or layer norms, as suggested by PaLM (63). We have observed through the literature and in internal experiments that these architecture changes improve the stability and performance of models.
A core architectural modification we make is the insertion of the Geometric Attention sub-layer in the first block of the network only (Appendix A.1.5, line 3).
Algorithm 2 transformer_block
Input: $x \in \mathbb{R}^{L \times d}, T \in S E(3)^{L}$
1: $s=\sqrt{\frac{36}{n_{\text {layers }}}}$
2: $x=x+s \cdot$ MultiHeadSelfAttention $(x) \quad \triangleright \mathbb{R}^{L \times d}$
3: $x=x+s$. geometric_mha $(x, T) \quad \triangleright \mathbb{R}^{L \times d}$
4: $x=x+s \cdot \operatorname{SwiGLUMLP}(x) \quad \triangleright \mathbb{R}^{L \times d}$
ESM3-small (1.4B) is a 48-layer network, while ESM3medium (7B) has 96 layers, and ESM3-large (98B) has 216 layers. We experimented with different width-to-depth ratios and observed higher returns for depth than width. Prior work also demonstrates that modalities like ours benefit more from deeper networks $(64,65)$. Detailed network specifications can be found in Table S1.
There are 7 unique input tracks to ESM3: (a) sequence (amino acid tokens), (b) structure coordinates, (c) struc- ture tokens, (d) 8-class secondary structure labels (SS8), (e) quantized solvent-accessible surface area (SASA) values, (f) function keyword tokens and (g) residue (InterPro) annotation binary features.
There are two additional tracks used during pre-training only: (h) per-residue confidence (pLDDT) and (i) averaged confidence (pLDDT). At inference time, these values are fixed, and these tracks are equivalent to adding a constant vector $z_{\text {plddt }}$.
Structure coordinates are parsed through the Geometric Attention and are not embedded.
For keyword-based function tokens, each of the eight integers per residue is converted to a "sub-embedding" (Appendix A.1.5.1 line 5), then concatenated to form the perresidue embedding (Appendix A.1.5.1 line 6). For InterPro residue annotations, the inputs are multi-hot. To create an embedding vector, we sum the embeddings for each of the "on" features (equivalent to the matrix-multiply on Appendix A.1.5.1 line 7).
The largest model, 98B has an additional taxonomy track detailed in Appendix A.1.9.2, only enabled in the final $30 \mathrm{~K}$ steps of pre-training.
The embeddings are all summed as input to the first layer in the network architecture.
Algorithm 3 encode_inputs
Input: $\mathrm{x}_{\text {inputs }}=$
$\left\{x_{\text {seq }}, x_{\text {structure }}, x_{\text {ss } 8}, x_{\text {sasa }}, x_{\text {func }}, x_{\text {res }}, x_{\text {plddt }}, x_{\text {avgplddt }}\right\}$
$z_{\text {seq }}=\operatorname{embed}\left(x_{\text {seq }}\right) \quad \triangleright \mathbb{R}^{L \times d}$
$z_{\text {structure }}=\operatorname{embed}\left(x_{\text {structure }}\right) \quad \triangleright \mathbb{R}^{L \times d}$
$z_{\mathrm{ss} 8}=\operatorname{embed}\left(x_{\mathrm{ss} 8}\right) \quad \triangleright \mathbb{R}^{L \times d}$
$z_{\text {sasa }}=\operatorname{embed}\left(x_{\text {sasa }}\right) \quad \triangleright \mathbb{R}^{L \times d}$
$h_{\text {func }, i}=\operatorname{embed}\left(\left[x_{\text {func }}\right]_{:, i}\right) \quad \triangleright \mathbb{R}^{L \times \frac{d}{8}}$
$z_{\text {func }}=\left[h_{\text {func }, 1}\left|h_{\text {func }, 2}\right| \ldots \mid h_{\text {func }, 8}\right] \quad \Delta \mathbb{R}^{L \times d}$
$z_{\text {res }}=x_{\mathrm{res}} W_{\text {res }} \quad \triangleright \mathbb{R}^{L \times d}$
$z_{\text {plddt }}=$ plddt_embed $\left(x_{\text {plddt }}, x_{\text {avgplddt }}\right) \quad \triangleright \mathbb{R}^{L \times d}$
return $z_{\text {seq }}+z_{\text {plddt }}+z_{\text {structure }}+z_{\text {ss } 8}+z_{\text {sasa }}+z_{\text {func }}+z_{\text {res }}$
We use a regressionhead to take in $d$ dimensional last layer hidden features and produce $c{\text {track }}$-dimensional logits for each of the tracks, where $c{\text {track }}$ corresponds to the size of the vocabulary per track. Note that for the keyword function tokens, we produce $c{\text {func }} \times 8$ logits, and softmax over each of the 8 independently when calculating the loss.
| Params | $n{\text {layers }}$ | $d{\text {model }}$ | $d_{\text {head }}$ | Context
length | Learning Warmup
rate | Batch
steps
size in
tokens | Num
steps | Total
tokens | FLOPs | |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| 1.4B | | | | | | | | | | |
| 1.4B | 48 | 1536 | 64 | 2048 | $4.0 \mathrm{e}-4$ | $5 \mathrm{~K}$ | $1,572,864$ | $50 \mathrm{~K}$ | $\sim 80 \mathrm{~B}$ | $6.72 \times 10^{20}$ |
| 7.7B | 96 | 2560 | 64 | 2048 | $4.0 \mathrm{e}-4$ | $5 \mathrm{~K}$ | $1,572,864$ | $200 \mathrm{~K}$ | $\sim 320 \mathrm{~B}$ | $2.7 \times 10^{21}$ |
| 98.5B | 216 | 6144 | 128 | 2048 | $2.5 \mathrm{e}-4$ | $5 \mathrm{~K}$ | $3,932,160$ | $140 \mathrm{~K}$ | $\sim 550 \mathrm{~B}$ | $2.47 \times 10^{22}$ |
Table S1. Parameter details for different model configurations.
Algorithm 4 regression_head
Input: $x \in \mathbb{R}^{\cdots \times d}$
1: $z=\operatorname{proj}_{\text {in }}(x)$
2: $z=\operatorname{GeLU}(z)$
3: $z=\operatorname{LayerNorm}(z)$
4: $z=\operatorname{proj}_{\text {out }}(z)$
return $z$
Except for structure coordinates, we output predictions for each of the tracks detailed in Appendix A.1.5.1: (a) sequence, (b) structure tokens, (c) SS8, (d) quantized SASA, (e) function keyword tokens and (f) residue (InterPro) annotation binary features.
Except for the multi-hot residue annotations, all other tracks are predicted as a categorical distribution over possible tokens.
As mentioned in Appendix A.1.5.1, ESM3 processes structural information in two independent ways:
Geometric Attention Described in Algorithm 6, this leverages fine-grained 3D information via conditioning on exact coordinates. We find that conditioning on coordinates is critical to good inverse folding performance. Coordinates are only used as model inputs.
Structure Tokens Described in Appendix A.1.7, structure tokens enable faster learning due to rich local neighborhood semantics being compressed into tokens. Structure tokens are generally used as model outputs.
Geometric attention enables high-throughput encoding of protein structures. Protein backbone structure can be represented by the relative distance and orientation of frames defined by each residue's backbone coordinates. Reasoning over the relative orientation of frames is important to capture the local backbone geometry when only partial structure is provided. Geometric attention is an $S E(3)$ invariant allto-all attention mechanism which reasons over the relative distances and orientations of all defined frames in the input (Fig. S2). Because this attention operation can be realized using the same computational primitives as attention, it is readily scalable.
We first provide an overview of frames, and then describe how geometric attention uses them:
Frames are representations that encapsulate the 3D positional and rotational information of residue backbones and sidechains in a protein structure. We use a formulation similar to Ingraham et al. (66). Each frame $T \in S E(3)$ consists of a rotation matrix $\mathbf{R} \in S O(3)$ and a translation vector $\mathbf{t} \in \mathbb{R}^{3}$
Definition: A frame $T_{i}$ for residue $i$ is defined as:
$$ T{i}=\left[\begin{array}{cc} \mathbf{R}{i} & \mathbf{t}{i} \ \mathbf{0}{1 \times 3} & 1 \end{array}\right] \in S E(3) $$
where $\mathbf{R}{i} \in S O(3)$ and $\mathbf{t}{i} \in \mathbb{R}^{3}$.
Rotation Matrix: The rotation matrix $\mathbf{R}{i}$ for residue $i$ is composed of three 3-dimensional vectors $\left[\hat{x}, \hat{e}{1}, \hat{e}_{2}\right]$ :
This matrix rotates vectors to a local coordinate system where the $N-C_{\alpha}-C$ plane for the corresponding residue spans the $x y$ plane.
Translation Vector: The translation vector $\mathbf{t}{i}$ specifies the position of the residue's $C{\alpha}$.
Transformation: To transform a point $\mathbf{p} \in \mathbb{R}^{3}$ from the local frame of residue $i$ to the global coordinate system, the following equation is used:
$$ \mathbf{p}{\text {global }}=T{i}(\mathbf{p})=\mathbf{R}{i} \mathbf{p}+\mathbf{t}{i} $$
Inverse Transformation: To transform a point $\mathbf{p}_{\text {global }} \in$ $\mathbb{R}^{3}$ from the global coordinate system back to the local frame of residue $i$, the following equation is used:
$$ \mathbf{p}=T{i}^{-1}\left(\mathbf{p}{\text {global }}\right)=\mathbf{R}{i}^{-1}\left(\mathbf{p}{\text {global }}-\mathbf{t}_{i}\right) $$
Figure S2. Geometric attention. Geometric attention is an $\mathrm{SE}(3)$ invariant all-to-all attention mechanism where the attention score matrix is a weighted sum of two terms: (1) the pairwise distances between queries and keys rotated and translated by their respective backbone frames, and (2) the pairwise dot products between queries and keys rotated by their respective backbone frames. This attention mechanism encodes structural information with throughput comparable to the standard attention operation in transformers.
To create frames, all we require is a translation vector $\vec{t}$, and two vectors $\vec{x}$ and $\vec{y}$ defining the local $x y$ plane after conversion to global coordinates, from which the frame $T$ can be calculated with the standard Gram-Schmidt algorithm:
Algorithm 5 gram_schmidt
Input: $\vec{t} \in \mathbb{R}^{L \times 3}, \vec{x} \in \mathbb{R}^{L \times 3}, \vec{y} \in \mathbb{R}^{L \times 3}$
$: \hat{x}=\frac{\vec{x}}{\|\vec{x}\|}$
$\vec{e}_{1}=\vec{y}-(\hat{x} \cdot \vec{y}) \hat{x}$
$\hat{e}_{1}=\frac{\vec{e}_{1}}{\left\|\vec{e}_{1}\right\|}$
$\hat{e}_{2}=\hat{x} \times \hat{e}_{1}$
$R=\left[\hat{x}, \hat{e}_{1}, \hat{e}_{2}\right] \quad \triangleright S O(3)^{L}$
$T=\left[\begin{array}{cc}R & \vec{t} \\ 0_{1} \times 3 & 1\end{array}\right] \quad \triangleright S E(3)^{L}$
return $T$
We construct frames such that the $C_{\alpha}$ is at the origin of the frame $(\vec{t}), C$ on the negative x-axis $(-\vec{x})$, and $N$ is on the $x y$-plane.
Algorithm 6 details the Geometric Self-Attention layer. It can be efficiently implemented using similar ideas as FlashAttention (33). It is used twice in our system: in the VQ-VAE encoder for structure tokens (Appendix A.1.7.1), and in the first layer of ESM3.
Unlike regular self-attention, which only operates on perresidue embeddings, Geometric Attention incorporates the per-residue frames $T$ to integrate geometric information in a rotation and translation invariant way. The process of forming the attention matrix $A$ is as follows:
(a) Convert to Global Rotational Frame: We convert each of the vectors in $Q{r}, K{r}, V$ from their local frame (where the $x y$ plane is the $N-C{\alpha}-C$ plane for each residue) to a global rotational frame (where the $x y$ plane is aligned for all residues) by applying $\mathbf{R}{i}$ (Algorithm 6, lines 3, 4).
(b) Convert to Global Distance Frame: We convert each of the vectors in $Q{d}, K{d}$ from their local frame to a global frame by applying $T_{i}$ (Algorithm 6 , lines 5, 6).
Directional Attention: The pairwise, per-head $h$ rotational similarity $R$ between keys $i$ and queries $j$ is calculated using the dot product $[R]{i, j, h}=\frac{1}{\sqrt{3}}\left[q{r}\right]{i, h,:}$. $\left[k{r}\right]_{j, h,:}$ This is equivalent to the cosine distance between projected points.
Distance Attention: The pairwise, per-head $h$ distance similarity $D$ between keys $i$ and queries $j$ is computed using the $L{2}$ norm of the difference $[D]{i, j, h}=$ $\frac{1}{\sqrt{3}}\left|\left[q{r}\right]{i, h,:}-\left[k{r}\right]{j, h,:}\right|_{2}$.
Scale Factor: $R$ and $D$ are scaled per-head with learned scalars $\left[\bar{w}{r}\right]{h}$ and $\left[\bar{w}{d}\right]{h}$, respectively, where $\bar{w}{r}, \bar{w}{d} \in \mathbb{R}^{h}$. We use the softplus function to transform weights into $[0, \infty)^{h}$. This scaling allows certain heads to specialize in attending via distance or directional attention.
Algorithm 6 geometric_mha
Input: $X \in \mathbb{R}^{L \times d}, T \in S E(3)^{L}$
$Q_{r}, K_{r}, Q_{d}, K_{d}, V=\operatorname{Linear}(X) \quad \triangleright\left(\mathbb{R}^{L \times h \times 3}\right)_{\times 5}$
$\left(\mathbf{R}_{i}, \mathbf{t}_{i}\right)=T_{i} \quad \triangleright\left(S O(3)^{L}, \mathbb{R}^{L \times 3}\right)$
$\left[Q_{r}\right]_{i, h,:}=\mathbf{R}_{i}\left(\left[Q_{r}\right]_{i, h,:}\right) \quad \triangleright \mathbb{R}^{L \times h \times 3}$
$\left[K_{r}\right]_{i, h,:}=\mathbf{R}_{i}\left(\left[K_{r}\right]_{i, h,:}\right)$
$\triangleright \mathbb{R}^{L \times h \times 3}$
$\left[Q_{d}\right]_{i, h,:}=T_{i}\left(\left[Q_{d}\right]_{i, h,:}\right) \quad \triangleright \mathbb{R}^{L \times h \times 3}$
$\left[K_{d}\right]_{i, h,:}=T_{i}\left(\left[K_{d}\right]_{i, h,:}\right) \quad \triangleright \mathbb{R}^{L \times h \times 3}$
$7:[R]_{i, j, h}=\frac{1}{\sqrt{3}}\left[q_{r}\right]_{i, h,:} \cdot\left[k_{r}\right]_{j, h,:} \quad \triangleright \mathbb{R}^{L \times L \times h}$
8: $[D]_{i, j, h}=\frac{1}{\sqrt{3}}\left\|\left[q_{r}\right]_{i, h,:}-\left[k_{r}\right]_{j, h,:}\right\|_{2} \quad \triangleright \mathbb{R}^{L \times L \times h}$
9: $A=\operatorname{softplus}\left(\bar{w}_{r}\right) R-\operatorname{softplus}\left(\bar{w}_{d}\right) D \quad \triangleright \mathbb{R}^{L \times L \times h}$
$A=\operatorname{softmax}_{j}(A)$
$[V]_{i, h,:}=\mathbf{R}_{i}\left([V]_{i, h,:}\right)$
$O=A \cdot V \quad \triangleright \mathbb{R}^{L \times h \times 3}$
$[O]_{i, h,:}=\mathbf{R}_{i}^{-1}\left([O]_{i, h,:}\right)$
$X=X+\operatorname{Linear}(O)$
$\triangle \mathbb{R}^{L \times d}$
Each residue is associated with one of 4,096 structure tokens ( +4 special tokens), designed to provide a rich, learned representation of its local neighborhood. The tokens are generated with a VQ-VAE encoder, with a corresponding decoder to enable decoding of generated tokens back to $3 \mathrm{D}$ coordinates.
The VQ-VAE encoder $f{\text {enc }}$ consists of two geometric attention blocks (Transformer blocks, but self-attention replaced with geometricmha) with an embedding width of 1024 and 128 geometric heads per geometric attention layer.
The VQ-VAE encoder reasons over the backbone frames and the relative sequence position of residues in the local structure. Relative sequence positions are encoded through a learned positional embedding. Sequence positions are determined relative to the query residue (i.e., if the query residue has residue index 56 , then the residue in index 58 has a +2 sequence position). Relative sequence positions are clamped to $+/-32$ before encoding, meaning long-range contacts share sequence positional embeddings. Relative sequence positional embeddings define the initial encoder state $N$, and has shape $L \times 16 \times d$ (Algorithm 7, line 4). Note that this means the input to the VQ-VAE encoder is purely structural: no sequence (amino acid), function or other information is used here. Furthermore, each neighborhood is processed completely independently; for each residue, the encoder only uses the information of its 16 nearest neighbors.
Geometric attention blocks operate similar to Transformer blocks in that they transform a state according to an attention operation ( geometricmha ) and feedforward network (SwiGLU MLP). As such, the output has the same shape as the input. In this case, this means that the encoder outputs 16 latents per residue. However, we want to learn a single token, i.e., a single latent per residue, hence we take the embedding corresponding to the query residue position $N{:, 0,:}$.
The process of generating structure tokens (Algorithm 7) from the full 3D coordinates of the protein then is as follows:
Input: $x_{C_{\alpha}} \in \mathbb{R}^{L \times 3}, T \in S E(3)^{L}$
1: $N_{\mathrm{idx}}=\operatorname{knn}\left(x_{C_{\alpha}}\right) \quad \triangleright\{0 . . L-1\}^{L \times 16}$
$: T_{\mathrm{knn}}=T\left[N_{\mathrm{idx}}\right] \quad \triangleright S E(3)^{L \times 16}$
$\Delta i=\operatorname{clamp}\left(N_{\mathrm{idx}}-i,-32,32\right)$
$N=\operatorname{embed}(\Delta i)$
$\Delta \mathbb{R}^{L \times 16 \times d}$
5: $N=f_{\text {enc }}\left(N, T_{\mathrm{knn}}\right)$
$\triangle \mathbb{R}^{L \times 16 \times d}$
6: $z=\operatorname{Linear}\left(N_{:, 0,:}\right) \quad \triangleright \mathbb{R}^{L \times d^{\prime}}$
7: $z=$ quantize $(z) \quad \triangleright\{0 . .4095\}^{L \times 16}$
quantize transforms the $L$ latents into $L$ discrete tokens. Since the VQ-VAE was initially proposed (67), numerous approaches and tricks have been developed to address issues with poor codebook utilization and unstable training. We chose to learn the codebook as an exponential moving average of encoder outputs (67-69). To improve codebook utilization, unused codes are re-initialized to encoder outputs.
To improve training and inference efficiency, we encode all local structure graphs within a protein in parallel. In practice, this means that given a batch of $B$ proteins with average sequence length $L$, then the inputs to the structure encoder will have shape $B L \times 16 \times d$.
While the encoder independently processes all local structures in parallel, the decoder $f_{\text {dec }}$ attends over the entire set of $L$ tokens to reconstruct the full structure. It is composed using a stack of bidirectional Transformer blocks with regular self-attention.
As discussed in Appendix A.1.7.3, the VQ-VAE is trained in two stages. In the first stage, a smaller decoder trunk consisting of 8 Transformer blocks with width 1024, rotary positional embeddings, and MLPs is trained to only predict backbone coordinates. In the second stage, the decoder weights are re-initialized and the network size is expanded to 30 layers, each with an embedding dimension of 1280 ( $\sim 600 \mathrm{M}$ parameters) to predict all atom coordinates.
The exact steps to convert structure tokens back to 3D allatom coordinates using the decoder is provided in Algorithm 8 and detailed as follows,
Algorithm 8 structure_decode
Input: $z \in\{0 . .4099\}^{L \times 16}$
1: $z=\operatorname{embed}(z)$
$\triangle \mathbb{R}^{L \times d}$
2: $z=f_{d e c}(z)$
$\triangleright \mathbb{R}^{L \times d}$
3: $\vec{t}, \vec{x}, \vec{y}, \sin \theta, \overline{\cos \theta}=\operatorname{proj}(z) \quad \triangleright\left(\mathbb{R}^{L \times 3}\right)_{\times 3},\left(\mathbb{R}^{L \times 7}\right)_{\times 2}$
4: $T=$ gram_schmidt $(\vec{t},-\vec{x}, \vec{y}) \quad \triangle S E(3)^{L}$
5: $\sin \theta=\frac{\overline{\sin \theta}}{\sqrt{\sin ^{2}+\overline{\cos \theta}}} \quad \triangleright[-1,1]^{L \times 7}$
7: $T_{\text {local }}=$ rot_frames $(\sin \theta, \cos \theta) \quad \triangleright S E(3)^{L \times 7}$
8: $T_{\text {global }}=$ compose $\left(T_{\text {local }}, T\right) \quad \triangleright S E(3)^{L \times 14}$
9: $\vec{X}=T_{\text {global }}\left(\overrightarrow{X_{r e f}}\right) \quad \triangleright \mathbb{R}^{L \times 14 \times 3}$
When using a VQ-VAE to learn discrete representations which maximize reconstruction quality, it is common to train in the autoencoder in two stages (70). In the first stage, the encoder and codebook is learned with a relatively small and efficient decoder. In the second stage, the encoder and codebook are frozen and a larger or otherwise more computationally expensive decoder is trained to maximize reconstruction quality. We follow this two-stage training approach for the structure tokenizer.
The VQ-VAE is trained for $90 \mathrm{k}$ steps on a dataset of single chain proteins from the PDB, AFDB, and ESMAtlas. We use the AdamW optimizer (Loshchilov et al. 2017) with learning rate annealed from $4 \mathrm{e}-4$ according to a cosine decay schedule. Proteins are cropped to a maximum sequence length of 512. Five losses are used to supervise this stage of training. The geometric distance and geometric direction losses are responsible for supervising reconstruction of high quality backbone structures.
Additionally, a distogram and binned direction classification loss are used to bootstrap structure prediction but are ultimately immaterial to reconstruction. We have found that these structure prediction losses formulated as classification tasks improve convergence early in training. To produce these pairwise logits, we use a pairwiseprojhead, that takes $x \in \mathbb{R}^{L \times d}$ and returns logits $z \in \mathbb{R}^{L \times L \times d^{\prime}}$. It works as follows:
Algorithm 9 pairwise_proj_head
Input: $x \in \mathbb{R}^{L \times d}$
$q, k=\operatorname{proj}(x), \operatorname{proj}(x)$
$: \operatorname{prod}_{i, j,:} \operatorname{diff}_{i, j,:}=q_{j,:} \odot k_{i,:}, q_{j,:}-k_{i,:}$
$z=$ regression_head $([$ prod $\mid$ diff $]) \triangleright \mathbb{R}^{L \times L \times d^{\prime}}$
return $z$
Finally, an inverse folding token prediction loss (i.e., a crossentropy loss between predicted sequence and ground truth sequence) is an auxiliary loss used to encourage the learned representations to contain information pertinent to sequencerelated tasks.
The five losses are covered in detailed as follows:
Algorithm 10 backbone_distance_loss
Input: $\hat{X} \in \mathbb{R}^{L \times 3 \times 3}, X \in \mathbb{R}^{L \times 3 \times 3}$
: $\hat{Z}, Z=\operatorname{flatten}(\hat{X})$, flatten $(X) \quad \triangleright \mathbb{R}^{3 L \times 3}, \mathbb{R}^{3 L \times 3}$
$\left[D_{\text {pred }}\right]_{i, j}=\left\|[\hat{Z}]_{i,:}-[\hat{Z}]_{j,:}\right\|_{2}^{2} \quad \triangleright \mathbb{R}^{3 L \times 3 L}$
$[D]_{i, j}=\left\|[Z]_{i,:}-[Z]_{j,:}\right\|_{2}^{2} \quad \triangleright \mathbb{R}^{3 L \times 3 L}$
$E=\left(D_{\text {pred }}-D\right)^{2}$
$E=\min (E, 25)$
$l=\operatorname{mean}_{i, j}(E)$
$\triangle \mathbb{R}$
return $l$
(b) $C_{\alpha} \rightarrow C$
(c) $C \rightarrow N_{\text {next }}$
(d) $\mathbf{n}{C{\alpha}}=-\left(N \rightarrow C{\alpha}\right) \times\left(C{\alpha} \rightarrow C\right)$
(e) $\mathbf{n}{N}=\left(C{\text {prev }} \rightarrow N\right) \times\left(N \rightarrow C_{\alpha}\right)$
(f) $\mathbf{n}{C}=\left(C{\alpha} \rightarrow C\right) \times\left(C \rightarrow N_{\text {next }}\right)$
Compute the pairwise dot product, forming $D{\text {pred }}, D \in$ $\mathbb{R}^{6 L \times 6 L}$. Compute $\left(D{\text {pred }}-D\right)^{2}$, clamp the maximum error to 20 , and take the mean.
In algorithm form (with compute_vectors computing the six vectors described above):
Algorithm 11 backbone_direction_loss
Input: $\hat{X} \in \mathbb{R}^{L \times 3 \times 3}, X \in \mathbb{R}^{L \times 3 \times 3}$
$\hat{V}=$ compute_vectors $(\hat{X}) \quad \triangleright \mathbb{R}^{6 L \times 3}$
$V=$ compute_vectors $(X) \quad \triangle \mathbb{R}^{6 L \times 3}$
$\left[D_{\text {pred }}\right]_{i, j}=[\hat{V}]_{i,:} \cdot[\hat{V}]_{j,:} \quad \triangleright \mathbb{R}^{6 L \times 6 L}$
$[D]_{i, j}=[V]_{i,:} \cdot[V]_{j,:} \quad \triangleright \mathbb{R}^{6 L \times 6 L}$
$E=\left(D_{\text {pred }}-D\right)^{2}$
$E=\min (E, 20)$
$l=\operatorname{mean}_{i, j}(E) \quad \triangleright \mathbb{R}$
return $l$
(a) Unit vectors: Compute three vectors per residue from ground truth coordinates: $C{\alpha} \rightarrow C, C{\alpha} \rightarrow$ $N$, and $\mathbf{n}{C{\alpha}}=\left(C{\alpha} \rightarrow C\right) \times\left(C{\alpha} \rightarrow N\right)$, and normalize them to unit length.
(b) Dot Products: Compute pairwise dot products between each pair of vectors for all residues, forming $D \in[-1,1]^{L \times L \times 6}$. Bin the dot products into 16 evenly spaced bins in $[-1,1]$, forming classification labels $y \in{0 . .15}^{L \times L}$.
(c) Pairwise Logits: Pass the final layer representations of the decoder $h \in \mathbb{R}^{L \times d}$ through a pairwiseprojhead to obtain logits $z \in$ $\mathbb{R}^{L \times L \times 6 \times 16}$.
(d) Cross Entropy: Calculate cross-entropy loss using the labels $y$ from the ground truth structure and the logits $z$, and average over all $L \times L \times 6$ values.
i. Obtain the three vectors $N \rightarrow C{\alpha}, C{\alpha} \rightarrow C$, and $\mathbf{n}=\left(N \rightarrow C{\alpha}\right) \times\left(C{\alpha} \rightarrow C\right)$.
ii. Define the following scalars:
$$ \begin{aligned} a & =-0.58273431 \ b & =0.56802827 \ c & =-0.54067466 \end{aligned} $$
iii. Compute the location of $C_{\beta}$ using the formula:
$C{\beta}=a \mathbf{n}+b\left(N \rightarrow C{\alpha}\right)+c\left(C{\alpha} \rightarrow C\right)+C{\alpha}$
(b) Pairwise $C{\beta}$ distances: Compute an $L \times L$ pairwise distance matrix of the $C{\beta}$, and bin them into one of 64 bins, with lower bounds $\left[0,2.3125^{2},(2.3125+0.3075)^{2}, \ldots, 21.6875^{2}\right]$, forming the labels $y \in{0 . .63}^{L \times L}$.
(c) Pairwise logits: Pass the final layer representations of the decoder $h \in \mathbb{R}^{L \times d}$ through a pairwiseprojhead to obtain the logits $z \in \mathbb{R}^{L \times L \times 64}$.
(d) Cross Entropy: Calculate the cross-entropy using the labels $y$ computed from the ground truth structure and the logits $z$, then average over all $L \times L$ values.
In the second stage of VQ-VAE training, the encoder and codebook are frozen and a new, deeper, decoder is trained. This second stage of training has multiple purposes. First, a larger decoder improves reconstruction quality. Second, augmented structure tokens from ESM3 are added to enable learning pAE and pLDDT heads. Third, we add sequence conditioning and train with all-atom geometric losses to be able to decode all-atom protein structures. Fourth, we extend the context length of the decoder to be able to decode multimers and larger single chain proteins.
Training data for stage 2 consists of predicted structures in AFDB and ESMAtlas, as well as single chain, multimer, and antibody-antigen complexes from the PDB. Sequence conditioning was added to the decoder via learned embeddings which are summed with structure token embeddings at the input to the decoder stack.
The structure token decoder was trained in three stages: $2 \mathrm{~A}$, 2B, 2C detailed in Table S2. The purpose of stage 2A is to efficiently learn decoding of all-atom structures. Enhanced training efficiency is achieved by keeping a short context length and omitting the pAE and pLDDT losses, which are both memory-consuming and can be in competition with strong reconstruction quality. In stage $2 \mathrm{~B}$, we add the pAE and pLDDT losses. These structure confidence heads cannot be well-calibrated unless structure tokens are augmented such that ESM3-predicted structure tokens are within the training distribution. To this end, for stages $2 \mathrm{~B}$ and $2 \mathrm{C}$ we replace ground truth structure tokens with ESM3-predicted structure tokens $50 \%$ of the time. In stage $2 \mathrm{C}$, we extend context length to 2048 and upsample experimental structures relative to predicted structures.
(a) Compute a pairwise distance matrix per residue from the 3D coordinates of each atom in atom14 representation, resulting in $\mathbb{R}^{L \times 14 \times 14}$.
(b) Mark atoms less than $2 \AA$ apart (excluding self) as covalent bonds.
(c) Filter to keep atoms with at least 2 covalent bonds, keeping only the first 2 bonds per atom, with ordering determined by the atom 14 representation.
(d) For each selected atom, compute a normal (zaxis) vector to the plane spanned by its two covalent bonds, resulting in three vectors per selected atom.
(e) Randomly subsample to 10,000 vectors per protein if the number exceeds 10,000 , ensuring the same vectors are sampled in both predicted and ground truth structures.
(f) Compute all-to-all pairwise dot products, forming $D{\text {pred }}, D \in \mathbb{R}^{n \times n}$. Compute $\left(D{\text {pred }}-D\right)^{2}$, clamp the max to 20 , and take the mean.
(a) Compute the pairwise distances between residues in both the predicted and ground truth structures, resulting in distance matrices $D_{\text {pred }}$ and $D \in \mathbb{R}^{L \times L}$.
(b) Calculate the differences $\left(D_{\text {pred }}-D\right)$.
(c) Bin these differences into 64 bins, generating classification targets for the logits.
(d) Compute the loss using cross-entropy between these targets and the logits.
Computing pAE: Multiply probabilities by bin centers and sum to obtain the expected positional error per residue pair, with values $\in[0.25,31.75]$.
Computing pTM: Additionally, the pairwise logits are used to compute the pTM (Predicted Template Modeling) score, as follows:
(a) Compute $f_{d}$ for sequence length $L$ as:
$$ \begin{aligned} d{0} & =1.24 \cdot(\max (L, 19)-15)^{\frac{1}{3}}-1.8 \ f{d} & =\frac{1}{1+\left(\frac{\text { bins }}{d_{0}}\right)^{2}} \end{aligned} $$
(b) Compute $\mathrm{pTM}$ using previously computed probabilities $p$ :
$$ \mathrm{pTM}=\max {i}\left[\frac{1}{L} \sum{j}\left(\sum{\text {bin }}[p]{i, j, \text { bin }}\left[f{d}\right]{\text {bin }}\right)\right] $$
We evaluate the reconstruction quality of the structure tokenizer after stage 1 and stage 2 of training using a set of CAMEO, CASP14, and CASP15 proteins taken after the training cutoff date (Fig. S3). Both decoders consistently reach RMSD $<1 \AA$, LDDT-CA $>0.98$. The retraining of the structure token decoder results in substantial improvements in reconstruction quality across all test sets. The stage 2 decoder, trained with an all-atom reconstruction loss and a sequence input, achieves strong all-atom reconstruction as well (Fig. S3C). We also visualize a random sample of backbone reconstructions on the CAMEO test set (Fig. S4A). Looking at the proteins with worse reconstruction quality, we find that long regions with few tertiary contacts, disordered regions, and unresolved coordinates
| Stage | Steps | All-atom
geometric
losses | pAE
pLDDT
losses | and
with ESM3-
predicted
tokens | Data mixture
length | |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| 2A | $90 \mathrm{k}$ | $\checkmark$ | $X$ | $X$ | 512 | Roughly uniform sampling of pre-
dicted and experimental structures |
| 2B | $20 \mathrm{k}$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 512 | Roughly uniform sampling of pre-
dicted and experimental structures |
| 2C | $30 \mathrm{k}$ | $\checkmark$ | $\checkmark$ | | 2048 | Upsampling experimental structures |
Table S2. Training details for stage 2 training of an all-atom structure token decoder.
can lead to inaccurate global orientation of structural elements, while local structure reconstruction remains largely error-free (Fig. S4B). This behavior can be explained by the fact that the tokenizer relies on tertiary contacts to resolve the global orientation of a residue.
We also investigate the vocabulary learned by the structure tokenizer by visualizing the local neighborhoods which map to the same learned structure token. We find that many structure tokens encode semantically coherent sets of local neighborhoods (Fig. S5A). However, a number of tokens appear to represent multiple local neighborhoods (Fig. S5B). While the meaning of a single token may be ambiguous, the high-fidelity reconstruction quality from the decoder suggests that it is able to disambiguate given surrounding context in the full set of structure tokens.
Fig. S6 indicates that pLDDT and pTM are well-calibrated. We assess the calibration of the structure confidence heads on the CAMEO test set using structure tokens predicted by ESM3 7B. Most predictions for pLDDT lie along the diagonal, though there is a small bias towards more confident predictions. As pTM is a pessimistic estimator of the TMscore, we find that pTM is biased downwards. Anecdotally, we also find that pLDDT can be poorly calibrated for some generated sequences, particularly in alpha helical regions where it can be an overestimate.
ESM3 processes annotations of functional characteristics of proteins through two tracks: function tokens, and residue annotations. Both support input conditioning and output heads for generation. Appendix A.1.5.1 outlines how tokens are processed into the network: we further describe the creation of the tokens themselves in this section.
Function tokens are a dense semantic representation of functional characteristics of proteins derived from free-text descriptions of the InterPro and Gene Ontology (GO) terms at each residue. At training time, function tokens are produced from each protein's InterPro annotations by a multi-step process illustrated in Fig. S7. At a high level:
The result is a sequence of 8 tokens each ranging in value from 0 to 255 per-residue. We reserve a special token
B
C
Figure S3. Structure tokenizer reconstruction quality. Reconstruction quality of the structure tokenizer after stage 1 and stage 2 of VQ-VAE decoder training evaluated on temporally held out CAMEO, CASP14, and CASP15. (A) Reconstruction LDDT-CA. (B) Reconstruction backbone RMSD. (C) All-atom reconstruction RMSD from the stage 2 decoder which additionally receives sequence input.
time, we can produce per residue vectors using the process described, or directly creating a TF-IDF vector with keywords.
During pre-training we use the corrupted versions of the function tokens at input, predicting the un-corrupted version function tokens at positions which have been masked. $90 \%$ of the time, the entire input is replaced with $<$ mask $>$. The other $10 \%$ of the time, we replace all 8 tokens of selected residue with a $<$ mask $>$, with the per-residue selection probability sampled from a cosine masking schedule per protein. The model has an output head which predicts each of the 8 function tokens in positions with $<$ mask $>$ as input, and is trained with a categorical cross entropy loss
Function tokenization offers several key advantages as compared to simpler approaches for example using a dedicated InterPro tag vocabulary. Encoding functional annotations with a generic functional keyword vocabulary enables flexible prompting of the model at test time, by combinations of keywords that were not encountered during training time. This enhances the programmability of ESM3 in designing novel proteins with not previously observed functional characteristics.
Function tokenization can also be viewed through the lens of data compression. This choice of representation reduces the input/output space from all possible InterPro combinations which would naively be represented by $35 \mathrm{k}$ bits, to a reduced space of 8 tokens $\times 8$ bits $/$ token $=64$ bits. This also affords significant memory saving during pre-training by eliminating the need to perform multi-class multi-label binary classification.
ESM3 is trained to predict all 8 function tokens, each spanning 256 possible values. To extract interpretable predictions of protein function from ESM3 we decode the predicted function tokens into function keywords using a seperately trained function token decoder.
We train a 3-layer transformer model to learn the inverse map of the function tokenization process. The model takes as input the 8 function tokens representing the locality sensitive hash of function keywords. It outputs for each residue the binary-classification predictions predicting the presence of each function keyword, as well as predicting InterPro annotations from which the keywords originate. We find that unpacking the 8-bit LSH tokens into single-bit tokens improves training dynamics of the function token decoder. We train the function token decoder offline using combinations of InterPro tags from the UniRef annotated proteins. Since the function token vocabulary is fixed the decoder is applied identically across different ESM3 model sizes.
To evaluate ESM3's performance in predicting protein function, we compute Average Precision, a standard measure of information retrieval, using the validation set of proteins from the UniRef and their associated InterProScan function annotations. We present results in Fig. S8.
Figure S4. Visualization of structure tokenizer backbone reconstructions. (A) A random sample of reconstructions from the structure tokenizer on the CAMEO test set. The vast majority of structures have near perfect backbone reconstruction (B) A selection of the worst reconstructions in CAMEO. Long stretches of disordered regions, a lack of tertiary contacts, and unresolved coordinates can lead to inaccurate global orientation of structural elements, while local structure reconstruction remains largely error-free.
B
Figure S5. Visualization of local neighborhoods which map to the same learned structure token. The VQ-VAE encoder reasons over local structure neighborhoods (highlighted in red) which include the query residue and the 15 nearest neighbors in structure space. (A) Rows correspond to token indices 585,59 , and 3692 for top, middle, and bottom, respectively. Columns show different local structures mapping to the same token. (B) Some tokens represent multiple types of local neighborhoods. All local neighborhoods in B map to the same codebook index 3276. While the meaning of a single token may be ambiguous, the decoder is able to disambiguate given surrounding context in the full sequence of structure tokens.
Figure S6. pTM and pLDDT calibration. Calibration of the structure token decoder pLDDT and pTM (using ESM3 7B as the structure token prediction model) on the CAMEO test set.
Figure S7. Schematic of function tokenization. The set of InterPro and GO descriptions of protein function are vectorized by a TF-IDF model and then hashed by a locality sensitive hash to produce 8 tokens each containing 8 bits.
Figure S8. Function prediction benchmarking results. Mean Average Precision (mAP) for function keyword prediction. Predictions and labels are compared on a per-position basis to evaluate the model's ability to localize site-specific functional attributes by keywords such as "active site". We report mAP for the full keyword set (red) with a "micro" average because many keywords have few or no labels in the validation set. To report a "macro" average mAP we compute mAP for each of the top 1,000 most prevalent keywords in our evaluation set (discarding uninformative keywords such as "the") and report a uniform average (blue). $95 \%$ confidence intervals are shown by shading.
Residue annotations label a protein's sites of functional residues with a vocabulary of 1474 multi-hot labels emitted by InterProScan. To gather this data, we run InterProScan with databases (SFLD, CDD, PIR) on all cluster members in our UniRef and Mgnify datasets (seq-id 90 clustered). We take all unique residue annotation descriptions that occur in more than $1 \mathrm{k}$ proteins across all of UniRef90 and MGnify 90 , and deduplicate labels by punctuation and case insensitivity. We join these annotations into our UniRef, MGnify, AFDB, and ESMAtlas datasets for training.
As introduced in Appendix A.1.5.1, ESM3 has a track dedicated to processing residue annotations that supports input conditioning, and an output head for generation. The residue annotation labels for a protein are tokenized into a sequence of token-sets in length equal to the protein. At each position there is an unordered set of tokens representing the residue annotations present at that position. The tokens are input to ESM3 first through an embedding lookup followed by a sum over embeddings. The permutation invariance of the sum retains that the labels are represented to an unordered set as a model. The per-position embedding sums are then added onto the per-position sequence embedding before input into the first transformer block. Positions with no residue annotations are represented by a
As mentioned in Appendix A.1.5.1, ESM3 has two additional tasks that are only used during pre-training, and only used as input (we do not have output heads predicting these values). The first is a per-residue pLDDT: for ground truth PDB structures, these values are all 1; for AlphaFoldDB/ESMFold structures, we use the provided pLDDT. We also provide an averaged pLDDT across all the residues when structure is provided (1 otherwise), with the average calculated before any tokens are masked.
This information allows the model to distinguish between gold-standard structures and computationally predicted ones; at inference time, we set these to 1 throughout, with the goal of producing structures better than the computational predictions used to pre-train the model. The embedding itself is straightforward, with the pLDDT values first having a radial basis function, followed by a Linear layer applied to them:
Algorithm 12 rbf
Input: $x \in \mathbb{R} \cdots \times L, a \in \mathbb{R}, b \in \mathbb{R}, n \in \mathbb{Z}^{+}$
$: \Delta=\frac{b-a}{n-1} \quad \quad \triangle \mathbb{R}$
$c=[a, a+\Delta, a+2 \Delta, \ldots, a+(n-2) \Delta, b] \quad \triangleright \mathbb{R}^{n}$
$\sigma=\frac{b-a}{n} \quad \triangle \mathbb{R}$
$[z]_{\ldots, i, j}^{n}=\frac{1}{\sigma}\left([x]_{\ldots, i}-[c]_{j}\right) \quad \triangle \mathbb{R}^{\cdots} \times L \times n$
return $\exp \left(-z^{2}\right) \quad \triangleright \mathbb{R} \cdots \times L \times n$
Algorithm 13 plddt_embed
Input: $x_{\text {plddt }} \in[0,1]^{L}, x_{\text {argplddt }} \in[0,1]$
$\operatorname{rbf}_{\text {plddt }}=\operatorname{rb} f\left(x_{\text {plddt }}, 0.0,1.0,16\right) \quad \triangle \mathbb{R}^{L \times 16}$
$\mathrm{rbf}_{\text {avgplddt }}=\operatorname{rb} f\left(x_{\text {avgplddt }}, 0.0,1.0,16\right) \quad \triangle \mathbb{R}^{16}$
$z_{\text {avgplddt }}=\operatorname{Linear}(\mathrm{rbf}$ avgplddt $) \quad \triangle \mathbb{R}^{d}$
$z_{\text {plddt }}=$ Linear(rbf plddt $) \quad \triangle \mathbb{R}^{L \times d}$
$\left[z_{\text {plddt }}\right]_{i,:}=\left[z_{\text {plddt }}\right]_{i,:}+z_{\text {avgplddt }} \quad \triangleright \mathbb{R}^{L \times d}$
return $z_{\text {plddt }}$
The final 30,000 steps in the pre-training of the $98 \mathrm{~B}$ variant of ESM3 includes a track for processing the taxonomic and species classification of the organism from which the protein sequence originates. For each protein, the taxonomic and species classifications are concatenated to create a full taxonomic lineage. The list of terms is then tokenized using a vocabulary comprised of the top 40,000 taxonomic terms in the UniRef training dataset. At input, learned embeddings (dimension 768) for each term in the lineage are summed and projected by a learned linear projection to a single embedding of $d_{\text {model }}$. This low-rank embedding bag saves memory as compared to using full-dimension embeddings. This single embedding is then repeated across the length of the sequence and summed into the positional embeddings with all the other tracks. The linear projection is zero-initialized at the start of this stage of training to preserve model behavior, enabling continuation of pre-training with no degradation in performance.
In pre-training we apply random corruption to the taxonomic lineages and train ESM3 to reconstruct the full lineage by predicting dropped terms with a shallow MLP head trained on the final layer's representations. To corrupt the taxonomic lineage, we either drop all terms (probability 25\%) or drop a set of the most specific terms of the lineage of size chosen uniformly at random from 1 to the length of the lineage (probability $25 \%$ ). We also independently drop any taxonomic term with probability $10 \%$. The output head outputs binary classification logits over the full set of 40,000 taxonomic lineage terms, and is trained to predict the missing terms via a binary-cross entropy loss.
Since ESM3 is a bidirectional transformer capable of representing arbitrary factorizations of the joint probability in any order or combination of tracks, we have significant flexibility during inference: We can generate sequence, structure, or function conditioned on any or no combination of other tracks. We also have a choice of how much compute to apply during generation.
The usual inference strategy is to fix a prompt (which may be a combination of any of the tracks, either fully or partially specified) and choose a track for generation (which may have been partially specified). When predicting the tokens for the generation track, a number of strategies are possible. Two notable strategies Argmax decoding, which predicts all tokens in the generation track in a single forward pass of the model; this computation is $O\left(L^{2}\right)$ in the length of the protein and is extremely efficient. Iterative decoding, on the other hand, samples tokens one position at a time, conditioning subsequent predictions on those already sampled. The runtime for iterative decoding, comparable to slower algorithms such as ESMFold and AlphaFold, is $O\left(L^{3}\right)$ in the length of the protein.
Additionally, the number of decoding steps can be chosen at runtime. Argmax decoding is equivalent to decoding in one step, while iterative decoding is equivalent to decoding in $L$ steps. It is possible to select any number of decoding steps between these two extremes to find an optimal tradeoff between computation and accuracy for a particular use case. See Appendix A.3.4 for a case study in the case of structure prediction, in which the generation track is the structure tokens track.
When using iterative decoding, ESM3 further allows flexibility in choosing the next position to decode. We choose the position based off of the logits output of ESM3, and for the results of this paper utilize two strategies: entropy decoding, which chooses the position with the lowest entropy after softmax, or max logit decoding, which chooses the position with the maximum logit. To generate $k$ tokens in one pass, we rank by either entropy or max logit and take the top $k$ positions.
In the following algorithm, assume a single forward pass of ESM3 is a function $f$ of a prompt $x$, and that we can access the logits of a specific token track through a subscript; e.g. sequence logits would be $f_{\text {sequence }}(x) \in \mathbb{R}^{L \times 32}$. Furthermore, denote $\pi(\cdot ; z)$ as the probability distribution induced by the logits $z$, including an implicit softmax, and $T \in \mathbb{R}^{L}$ a temperature schedule.
Algorithm 14 generate from track
Input: $x_{\text {prompt }}, n_{\text {decode }} \in\{1 . . L\}, T \in \mathbb{R}^{n_{\text {decode }}}$
: $k=L / n_{\text {decode }} \quad \triangleright \#$ steps to decode at each step
for $s \in\left\{1 . . n_{\text {decode }}\right\}$ do
$z_{\text {logits }}=$ esm3_forward $\left(x_{\text {prompt }}\right) \triangleright z \in \mathbb{R}^{L \times c_{\text {track }}}$
$\left\{p_{1}, \ldots, p_{k}\right\}=$ CHOosePositions $(z)$
for $i \in\left\{p_{1}, \ldots, p_{k}\right\}$ in parallel do
$x_{i} \sim \pi\left(x ; z_{i} / T_{s}\right) \quad \triangle$ Sample $i$ with temp $T_{s}$
$x_{\text {prompt }}=\left\{x_{\text {prompt }}, x_{i}\right\} \quad \triangleright$ Update prompt
end for
end for
Please summarize this for an expert
AI: The documentation excerpt details the architecture, input processing, and inference strategies of the ESM3 model, a comprehensive Transformer-based generative model designed for understanding and generating protein sequences, structures, and functions. Key aspects include:
Architecture: ESM3 integrates multiple tracks (inputs) for conditioning, including sequence tokenization, structure representation, secondary structure, solvent-accessible surface area (SASA), function annotations, residue annotations, and confidence scores. These inputs are processed through a standard Transformer architecture with adjustments for improved performance. Notable enhancements include the use of Pre-LN, SwiGLU activation, and rotary embeddings. A distinctive feature is the Geometric Attention mechanism that processes 3D structural information by leveraging the special Euclidean group $SE(3)$ to maintain rotation and translation invariance. The model's depth and width vary across configurations (small, medium, large), with the largest having 216 layers.
Tokenization and Encoding: Each input feature (e.g., protein sequences, structures, SASA values) is tokenized into a form the model can process. This involves unique strategies for different types of inputs—for example, structure representations through a VQ-VAE encoder and function annotations through TF-IDF vectorization followed by locality-sensitive hashing (LSH).
Inference Strategies: ESM3 supports flexible inference approaches, allowing fixed or partial input to be provided across any combination of its input tracks. The generative modeling approach can be executed through Argmax decoding (efficient, single pass) or iterative decoding (sampling tokens position-by-position, computationally expensive). The model allows dynamically choosing the number of decoding steps and the strategy for selecting token positions during generation, offering a trade-off between computational demand and accuracy.
Model Inputs and Training Details: The input to the ESM3 model is a detailed representation encompassing multiple aspects of a protein (e.g., sequence, structure, function). Training involves sophisticated methods to handle each type of input, including corrupting sequences for prediction during training and leveraging binary-cross entropy for the residue annotations track.
Function and Residue Annotations Processing: Function characteristics are captured through dense representations from free-text descriptions linked to InterPro and GO terms, converted to a manageable format via TF-IDF and LSH, resulting in quantized function tokens. Residue annotations are treated similarly but are encoded as multi-hot vectors representing the presence or absence of various functional annotations per residue.
Overall, ESM3's architecture, enhanced by geometric attention and specialized tokenization processes, is designed to advance protein sequence, structure, and function generation by efficiently integrating and processing multi-faceted biological data. Its inference strategies further allow for customizable applications, balancing between speed and precision based on specific needs.
UniRef release 202302 is downloaded and parsed from the official UniRef website (71). MGnify90 version 202302 is downloaded and parsed from MGnify (35). All nonrestricted studies available in JGI on July 31st, 2023 are downloaded and concatenated into the JGI dataset (72). OAS, which includes over a billion antibody sequences from
80 studies, is downloaded and clustered at $95 \%$ sequence identity (36).
In all cases, data is clustered with mmseqs2 (73), with flags --kmer-per-seq 100 --cluster-mode 2 --cov-mode 1 -c 0.8 --min-seq-id $<$ seqid>.
In order to do cluster expansion, we separately cluster the dataset at the two levels, and perform a join to determine cluster member and cluster center based on IDs. We first sample a cluster center at the lower level, and then sample a sequence within the cluster at the higher level. As an example, for expansion of UniRef70 at $90 \%$, we first cluster UniRef at $70 \%$ sequence similarity using mmseqs linclust. Then, we cluster it separately at $90 \%$. Since each UniRef90 cluster center is by definition a UniRef70 cluster member, we filter out UniRef70 for all cluster members that are in the UniRef90 clusters. We can then drop all cluster centers without any members, which may occur due to the nondeterminism of clustering. This allows us to sample a UniRef70 center, and then a member within that cluster, of which each are $90 \%$ sequence similarity apart. For ease of dataloading, we additionally limit the number of data points within a cluster to 20 .
As data augmention we train a 200M parameter inverse folding model and use it to create additional training examples.
The inverse folding model uses the geometric attention layer for structure conditioning and output projection head for the sequence logits as ESM3. Unlike ESM3 the transformer stack alternates between blocks with geometric attention and standard attention. The model is trained on the sequence and structure pairs in PDB, AlphaFold-DB, and ESMAtlas, with the single training task of (and loss computed on) predicting sequence at the output given structure at the input. Model architecture and training methodology is otherwise substantially similar to ESM3.
This model is used to generate additional sequences corresponding to each structure in the training data for ESM3 ( 5 sequences per structure for ESMAtlas and AlphaFold$\mathrm{DB}, 64$ sequences per structure for the $\mathrm{PDB})$. When training ESM3, with $50 \%$ probability the original sequence and structure pair is presented to the model as a training example. The other $50 \%$ of the time one of these 5 sequences is paired with the structure as the training example seen by ESM3.
Functional labels are obtained from InterPro (38) and InterProScan (74), both version 95.0. All annotations for UniPro- tKB were downloaded from the InterPro website via the 'protein2ipr.dat.gz' file. InterProScan was applied to the entirety of MGnify 90 with flags --goterms --iprlookup --pathways --disable-precalc. The resultant values are taken as ground truth functional labels for model training.
We use all PDB chains, clustered by unique PDB ID and entity ID within the PDB structure. We filter to all structures deposited before May 1, 2020, determined by X-ray crystallography, and better than $9 \AA$ resolution. (37)
AlphaFoldDB is downloaded as the $v 4$ version specified on their website (4). We notice that structures with high pLDDT are disproportionately alpha helices. Therefore, we ensure globularity by measuring the number of long range ( $>12$ sequence distance) contacts in the chain. If this value is $<0.5 \mathrm{~L}$ with an $\mathrm{L}$ length protein, we omit it from our training set. We also filter out all proteins $<0.7$ pLDDT.
ESMAtlas is downloaded as version v0 and v2023_02. Similarly we use $\mathrm{a}<0.7$ pLDDT filter. We use a $0.7 \mathrm{pTM}$ cutoff as well to enforce globularity. High pTM structures tends to be more compact.
Structural data also includes any functional labels that exist for the corresponding sequence.
For solvent accessibility surface area, we use the ShrakeRupley rolling probe algorithm as implemented in biotite (75). This generates a set of real numbers, or a nan value when structural coordinates are not provided. Similarly, SS8 labels are generated using the mkdssp tool (76) and taken as ground truth labels.
In both cases, we use the set of high quality predicted structures in AlphaFoldDB and ESMAtlas. We split our datasets into structural and sequence data. Structural data is shown separately in order to weight the ratios of structural data (mostly synthetic) properly with the amount of sequence data (mostly real).
An oversight was that we did not manage to apply these augmentations to PDB. However, since PDB constituted a relatively small portion of our training data, and these structural conditioning tasks did not depend on precise sidechain positions, we reasoned that high confidence synthetic structures would perform equally well and annotation of PDB was not necessary.
We keep track of validation set performance on a set of held out sequences from each training set, UniRef, MGnify, and JGI. In order to properly hold out a sufficiently diverse set of validation proteins, we first sample 25000 proteins from each set. Then we use mmseqs easy-search to filter out proteins from this set with a $70 \%$ sequence identity threshold. We choose the set of proteins from our training set to be the "query" set, and the set of validation proteins as our "target" set for mmseqs. We use the flags --alignment-mode 3 -c 0.8 {cov-mode 0 --max-seqs 300 --max-accept 3 --start-sens 2 -s 7 --sens-steps 3.
This query is designed such that early stopping in mmseqs will not affect if we find a hit in the "query" training set.
Train purges are run to generate a list of blacklisted UniRef, MGnify, and JGI IDs, which are removed from the training set.
The dataset counts in Table S3 are computed after limiting the large clusters to 20 . The number of tokens are computed by multiplying the number of sequences with the average length of the dataset.
In order to compute the approximate number of sequences and tokens seen during training, we first compute the number of times the dataset is repeated at the cluster level. Given the the number of repeats, we know the expected number of unique samples seen when sampling with replacement is $n\left(1-\left(1-\frac{1}{n}\right)^{k}\right)$ with a cluster of size $n$ and $k$ items selected. Computing this on the size of each cluster and number of dataset repeats results in the approximate number of tokens we present as presented in Table S4. Our largest model is trained on all of this data, while our smaller models use a portion of it depending on the model's token budget.
In the masked generative framework, corruption is applied to each input to the model. To enable generation, the amount of noise applied to an input is sampled from a distribution with probability mass on all values between 0 and 1 .
We select various noise schedules for different tracks with several goals in mind. First, ESM3 should see all combinations of tracks as input and output, enabling it to generate and predict based on arbitrary inputs. Second, ESM3 should maintain a balance of strong representation learning and high quality generations. Third, the type of inputs provided should be representative of what users would like to prompt the model with. In initial experimentation, we found that a fixed $15 \%$ noise schedule led to poor generation results, while a linear noise schedule where probability of each mask rate was constant led to good generation but poor representation learning results. We find a good trade-off between representation learning and generation by sampling the noise schedule from a mixture distribution. $80 \%$ of the time, the mask rate is sampled from a $\beta(3,9)$ distribution with mean mask rate $25 \%$. $20 \%$ of the time, the mask rate is sampled from a uniform distribution, resulting in an average overall mask rate of $30 \%$.
The noise schedules applied to each input are listed in Table S6. For the structure coordinate track, we also modify the noise to be applied as span dropping, as opposed to i.i.d over the sequence with $50 \%$ probability. This ensures that the model sees contiguous regions of masked and provided coordinates, which better mimics the types of inputs users may provide.
Along with applying noise to each track, we want to ensure ESM3 is able to perform well when some tracks are not provided at all (e.g. to perform structure prediction when no structure is provided as input). We enable this by wholly dropping out some tracks with varying probabilities, listed in Table S6.
We apply gaussian noise with standard deviation 0.1 to all coordinates the model takes as input.
An interesting use case of generative protein models involves conditioning on key structural information, such as an active site, and generating the sequence and structure of a protein that contains this information. It is possible to define an atomic coordination task as 3 residues which are mutually in contact in structure space ( $C \alpha-C \alpha$ distance $<6 \AA$ ), but are distant in sequence space ( $\geq 10$ positions apart) (23). Training on this conditioning may enable the model to better perform the type of atomic coordination required for active site sampling.
While this task will be sampled with some probability under the standard noise schedules, we also manually sample the task with $5 \%$ probability whenever a structure is available. If the task is sampled and a valid atomic coordination triplet is found, the structure coordinates for that triplet are shown to the model. For each residue in the triplet, the adjacent residues are also independently shown with $50 \%$ probability, which leads to a total size of between 3 and 9 residues. All other structure coordinates are masked. Normal masking is
| Dataset | Type | Clustering Level | Expansion Level | Tokens | Release | | :---: | :---: | :---: | :---: | :--- | :--- | | UniRef | Sequence | $70 \%(83 \mathrm{M})$ | $90 \%(156 \mathrm{M})$ | $54.6 \mathrm{~B}$ | $2023 _02$ | | MGnify | Sequence | $70 \%(372 \mathrm{M})$ | $90 \%(621 \mathrm{M})$ | $105.5 \mathrm{~B}$ | 202302 | | JGI | Sequence | $70 \%(2029 \mathrm{M})$ | - | $256 \mathrm{~B}$ | All non-restricted studies available on | | | | | | | July 30th, 2023. | | OAS | Sequence | $95 \%(1192 \mathrm{M})$ | - | $132 \mathrm{~B}$ | All sequences available on July 30th, | | | | | | | 2023. | | PDB | Structure | $-(203 \mathrm{~K})$ | - | $0.054 \mathrm{~B}$ | All chains available on RCSB prior to | | PDB Clustered | Structure | $70 \%(46 \mathrm{~K})$ | $100 \%(100 \mathrm{~K})$ | $0.027 \mathrm{~B}$ | May, 1st, 2020 | | AlphaFoldDB | Structure | $70 \%(36 \mathrm{M})$ | $90 \%(69 \mathrm{M})$ | $40.5 \mathrm{~B}$ | v4 | | ESMAtlas | Structure | $70 \%(87 \mathrm{M})$ | $90 \%(179 \mathrm{M})$ | $23.5 \mathrm{~B}$ | v0, v202302 |
Table S3. Pre-training dataset statistics. Includes number of tokens, release, and clustering level. Numbers are derived after dataset filtering.
| Dataset Name | Unique Samples(M) | Unique Tokens(M) | | :---: | :---: | :---: | | UniRef | 133 | 40,177 | | MGnify | 406 | 65,780 | | JGI | 2,039 | 265,070 | | OAS | 203 | 22,363 | | PDB | 0.2 | 55 | | AFDB | 68 | 20,510 | | ESMAtlas | 168 | 38,674 | | AFDB inverse folded | 111 | 33,300 | | ESMAtlas inverse folded | 251 | 57,730 | | Sequence | 3,143 | 484,441 | | Structure | 236 | 177,710 | | Annotation | 539 | 105,957 | | Total unique training tokens | | 768,109 |
Table S4. Pre-training unique token statistics. Broken down by token type and dataset type.
| Dataset | Inverse Folding | Function Labels | SASA | Secondary Structure | | ---: | :---: | :---: | :---: | :---: | | UniRef | $\checkmark$ | $\checkmark$ | - | - | | MGnify | $\checkmark$ | $\checkmark$ | - | - | | JGI | $x$ | $x$ | - | - | | OAS | $x$ | $x$ | - | - | | PDB | $x$ | $x$ | $x$ | $\mathbb{\checkmark}$ | | AlphaFoldDB | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | | ESMAtlas | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |
Table S5. Data augmentation and conditioning information applied to each dataset.
| Track | Noise Schedule | Dropout Prob | | :--- | :--- | :--- | | Sequence | betalinear30 | 0 | | Structure Tokens | cosine | 0.25 | | Structure Coordinates | cubic | 0.5 | | Secondary Structure (8-class) | square root | 0.9 | | SASA | square root | 0.9 | | Function Tokens | square root | 0.9 | | Residue Annotations | square root | 0.9 |
Table S6. Noise Schedules and Dropout Probabilities.
Figure S9. Visualization of noise schedules used. Left shows the probability density function of all noise schedules used. Right shows the betalinear30 distribution (which is drawn from $\beta(3,9)$ with $80 \%$ probability and a linear distribution with $20 \%$ probability) against a beta30 distribution (defined by $\beta(3,7)$ ) and a linear distribution.
applied to the other tracks.
Predicting and generating binding interfaces is another important task for generative protein models. To help with this capability, we add computational data augmentation that simulates the binding interface task.
We define a tertiary interface as one involving a long range contact $(C \alpha-C \alpha$ distance $<8 \AA, \geq 24$ sequence positions). When this task is sampled ( $5 \%$ probability whenever a structure is present), a long range contact is found, then the chain is split into two chains, each containing one side of the contact interface. Suppose the contacting positions are given by the indices $i, j$. Then the first chain will contain residues between [RANDINT $(1, i-3)$, RANDINT $(i+3, j-15)$ ], while the second chain will contain residues between [RANDINT $(i+15, j-3)$, RANDINT $(j+15, L)$ ]. This ensures there is always a residue gap between the two pseudochains. A chainbreak token "-" is inserted to represent the residue gap.
To encourage the model to learn to represent residue gaps using the chainbreak token, we introduce a task which randomly splits a single chain into multiple subchains.
First, a number of chains to sample is sampled from a geometric distribution with probability 0.9 , up to a maximum of 9 possible chains. If the number of chains sampled is 1 , no additional transformations are applied. A minimum separation of 10 residues between chains is defined. Sequence lengths of the chains along with gaps are sampled from a dirichlet distribution to maintain identically distributed sequence lengths for each chain. This transformation is applied to all samples.
In the case that multiple chains are provided to the model from either the interface sampling or pseudo-multimer augmentation tasks, we mask the geometric attention layer to prevent the model from attending to cross-chain coordinates. This simulates tasks where the structure of individual chains is known, but the interface is unknown.
We train all models using AdamW optimizer (77), with the following hyperparameters: $\beta{1}=0.9, \beta{2}=0.95$. We use a weight decay of 0.01 and gradient clipping of 1.0. We employ $5 \mathrm{~K}$ to $20 \mathrm{~K}$ warmup steps until reaching the maximum learning rate, and utilize a cosine decay scheduler to decay LR to $10 \%$ of the maximum learning rate by the end of training.
Our training codebase uses Pytorch. We use Pytorch's FSDP (78) implementation for data parallelism. We also use custom components from the TransformerEngine (79) library.
We have made several optimizations to increase the training speed of our models. For multi-head attention uses, we use the memory efficient implementation from the xformers library (80). We also save activations that are expensive to compute during training when necessary. We employ mixed precision training, utilizing FP8, BF16, and FP32 as needed based on accuracy requirements and kernel availability throughout our network.
Scaling ESM3 to 98 billion parameters with its novel architecture, multi-modal inputs, and low precision computation requirements poses significant training stability challenges. Our model is significantly deeper than its NLP counterparts, and literature has shown that deeper networks are harder to train due to attention collapse (81).
We observed training instability early in the architectural innovation phase, which we addressed through several changes. We apply layer normalization to the query and key vectors within the attention mechanism (82). We observe longer warm up helps (83). Another source of instability is the masking rate in pre-training tasks. We found that a very high masking rate is more likely to cause training divergences than a lower one, especially early in the training. Choosing a masking schedule biased towards lower mask rates improved both performance and training stability. Interestingly, the introduction of conditioning from other modalities also improves training stability, perhaps suggesting that stability is related to the degree of underspecification of a task.
An incorrectly set learning rate is another source of instability. To ensure the right balance between learning effectiveness and stability, we optimized the learning rate on smaller models and scaled it according to best practices as outlined in $(84,85)$. We find empirically that the initialization has a small effect on model stability, and the majority of stabilization can be gained from simply scaling the learning rate at the appropriate rate. By applying the rules in both width $-\mu \mathrm{P}$ and depth $-\mu \mathrm{P}$, we can simply scale the learning rate inversely proportional to the square root of the number of parameters, and find this results in stable training.
Following these modifications, we successfully trained our 98-billion-parameter model without any issues related to training instability.
We stage training to alter dataset composition, train on longer contexts that would be too expensive for the entire pre-training, or introduce features such as the taxonomy track (A.1.9.2.
Please analyze this paragraph from the point of view of an expert scientist and be critical of any omissions or possible sources of problems you see.
AI: The paragraph on "Staged Training" provides a brief overview on an adaptive training methodology used in the development of ESM3, which seems to involve adjusting the composition of the training dataset, extending the training on longer contexts, and incrementally incorporating additional features. While concise, this description leaves out critical details and raises several questions and potential concerns:
Lack of Specific Details: The paragraph does not provide concrete examples or specifics on how the dataset composition is altered. For instance, it does not mention what criteria are used to adjust the dataset composition or how these adjustments align with the training goals. This lack of specificity makes it difficult to assess the effectiveness and rationale behind the staged training approach.
Handling of Longer Contexts: Training on longer contexts is mentioned as a part of the staged approach but lacks detail about the implementation. How are these longer contexts selected or generated? What criteria define 'too expensive' in this context, and at what stage of training are longer contexts introduced? These missing details are crucial for understanding the scalability and computational efficiency of the training process.
Introduction of New Features: The incremental inclusion of features such as the taxonomy track is noted, but the paragraph does not explain how these features complement the existing data or the model's performance. Additionally, there is no mention of how the model architecture or training parameters are adjusted to accommodate these new features. The impact of these features on model stability, training time, and overall performance remains unclear.
Potential Overfitting Risks: The staged training approach could potentially increase the risk of overfitting, especially if the model is over-optimized on specific dataset compositions or features introduced in later stages. The paragraph does not discuss any strategies or techniques (e.g., regularization, validation checks) implemented to mitigate overfitting throughout the staging process.
Evaluation of Staged Training Effectiveness: There's no mention of how the effectiveness of staged training is assessed. Understanding whether staged training offers significant improvements over traditional methods is crucial. Metrics, comparative study results, or any form of evaluation that supports the adoption of staged training would provide much-needed validation for this approach.
Reproducibility Concerns: Absence of detailed methodology around staged training raises concerns about the reproducibility of the training process. Reproducibility is a cornerstone of scientific research, and the lack of detailed procedural information could hinder other researchers' ability to replicate the results or apply similar methodologies to different datasets or models.
In summary, while the concept of staged training seems promising for managing computational costs and progressively refining the model with complex features, the provided paragraph is overly vague and omits crucial details that would be necessary for a thorough scientific evaluation. Addressing these omissions would greatly enhance the understanding and potential utility of the staged training approach in large-scale model development.
ME: Please summarize for an expert
AI: The brief description on "Staged Training" within the development of ESM3 outlines an adaptive training strategy, emphasizing dataset adjustment, extended context training, and iterative feature inclusion. However, this summary is critically lacking in several areas necessary for thorough scientific evaluation and replication:
Specificity and Implementation Details: The summary fails to detail how dataset compositions are altered, criteria for longer context training, and the exact nature of features like the taxonomy track. This absence of specific implementation details and rationales prevents a clear understanding of the staged training approach’s mechanics and objectives.
Overfitting and Model Stability: There's no discussion on potential overfitting risks or measures to ensure model stability when introducing new data or features at various training stages. Without such considerations, the model’s generalizability and robustness remain uncertain.
Evaluation and Validation: The text lacks any mention of how the effectiveness of the staged training approach is gauged. Information on performance metrics, comparisons to other training methodologies, or any evaluative results is noticeably missing, making it difficult to assess the advantages or efficacy of this training strategy.
Reproducibility and Transparency: The omission of detailed methodologies hampers reproducibility. For scientific integrity and advancement, it is crucial to provide enough detail to allow others to replicate the results and explore the approach’s applicability to different models or datasets.
In essence, while staged training appears to hold promise for managing the computational demands and nuanced training requirements of large-scale models like ESM3, the document leaves significant gaps in methodological transparency, evaluation, and potential impacts on model performance and stability. Addressing these gaps is vital for a comprehensive understanding and scientific assessment of the approach.
ESM3 is both a generative model and a representation learning model that can be adapted for predictive tasks. In this section, we present benchmarking results for both capabilities.
ESM3 models are trained at three scales-1.4B, 7B, and 98B parameters-on approximately 75B, 560B, and 1.8T training tokens, respectively.
The ESM3 1.4B model, trained on 75B tokens and noted for its small size and speed, allows rapid iteration both during training and at inference. Optimal model size and number of training tokens are studied by extrapolating from a series of smaller runs, given a training compute budget, model architecture, and dataset characteristics $(19,21)$. After determining compute optimality for training, a variety of factors such as release frequency, amount of inference, ease of use, and usage patterns are also taken into account to determine the ideal number of tokens on which to train the model. To enable efficient inference for the benefit of the research community, we have trained two additional versions of ESM3 1.4B, named 1.4B Overtrained and 1.4B Open, which are trained on 300B tokens, far beyond their compute optimality for training.
In the following benchmarks for this section, unless otherwise noted, models are evaluated on a test set of 902 proteins whose structures are temporarily held out from the ESM3 training set. The proteins were sourced from the Continuous Automated Model EvaluatiOn (CAMEO) targets released from May 1, 2020 through Aug 1, 2023 (86).
For contact and structure prediction evaluations, we also evaluate on the CASP14 (71 proteins) and CASP15 (70 proteins) structure prediction benchmarks $(87,88)$. The CASP14 and CASP15 sets are obtained directly from the organizers.
The contact prediction model is a multilayer perceptron (MLP) head that operates independently over the representations of each amino acid pair, outputting the probability of contact between them. We use LoRA (89) for finetuning, which is a common alternative to full weight finetuning that uses much less memory while attaining strong performance. LoRA is applied to the base model for finetuning, and the MLP along with the LoRA weights are trained end-to-end using the cross-entropy loss with respect to the ground truth contact prediction map. For the ground truth, all residues at least 6 positions apart in the sequence and within an $8 \AA$ $\mathrm{C} \alpha$ - $\mathrm{C} \alpha$ distance are labeled as a contact. All models are trained with LoRA rank 4, batch size 64 and a learning rate of $1 \mathrm{e}-3$ for $10 \mathrm{k}$ steps on a mix of sequence and structure data from PDB, AlphaFold-DB, ESMAtlas, and OAS Predicted Structures. Data are sampled in a ratio of 1:3:3:0.03 from these datasets.
Table S7 shows the performance on each structural test set through the metric of precision at $\mathrm{L}(\mathrm{P} @ \mathrm{~L})$, which evaluates the precision of the top- $\mathrm{L}$ most confident predictions, where $\mathrm{L}$ is the length of the protein. The smallest ESM3 model, with 1.4B parameters, achieves a $\mathrm{P} @ \mathrm{~L}$ of $0.76 \pm 0.02$ on the CAMEO test set, which is higher than the $3 \mathrm{~B}$ parameter ESM2 model $(0.75 \pm 0.02)$. Furthermore, it trains on an order of magnitude less compute during pre-training ( $6.72 \times$ $10^{20}$ FLOPS vs. $1.8 \times 10^{22}$ FLOPS), demonstrating the benefits of multimodal pre-training.
ESM3 can directly predict protein structures without additional finetuning by first predicting structure tokens, then decoding these tokens into coordinates. When predicting structure tokens, we follow the strategy outlined in Appendix A.1.10 and test both argmax decoding and full iterative decoding.
For more difficult datasets, such as CASP14 and CASP15, iterative decoding has an outsized impact (see Table S8), whereas for easier datasets like CAMEO, argmax prediction is sufficient. On both the CAMEO and CASP15 datasets, argmax prediction for the 7B model is comparable to ESMFold, and iterative decoding with ESM3 98B closes the gap between ESMFold and Alphafold2. Structure prediction scaling curves as a function of training compute, are provided in Fig. S10
The conditional likelihood of an output given a prompt serves as a proxy for the generative capabilities of a model. Fig. S11 and Table S9 evaluate the performance of ESM3 as a conditional generative model, using its negative log likelihood (NLL) on the test set. For each track - sequence, structure, function, SASA, and secondary structure - NLL is evaluated both unconditionally and conditioned on each of the other tracks.
Figure S10. Scaling curves for structure prediction. Error bars are single standard deviations.
Unlike, for example, an autoregressive model, ESM3 is a generative model over masking patterns, so is trained to predict tokens given any masking pattern. The NLL of a sample under ESM3 is given by $\frac{1}{L!} \sum{o \in \mathbb{O}} \frac{1}{L} \sum{i=1}^{L} \log p\left(x{o{i}} \mid x{o{1}}, \ldots, x{o{i-1}}\right)$, where $O$ is the set of all decoding orders with normalization constant $Z=\frac{1}{L!}$. This computation is intractable (as the set of all decoding orders is exponential in length of a protein), but can be approximated by sampling a single decoding order $o$ for each $x$ in our dataset. At each step teacher forcing is used to replace the masked token with the ground truth token and report the mean NLL over the output tokens.
There are many straightforward relationships in this data. For example, the unconditional NLL (Fig. S11, black lines) is always higher than conditional, and conditioning on full $3 \mathrm{D}$ structure reduces the loss on secondary structure prediction to nearly zero (1.4B: $0.24,7 \mathrm{~B}: 0.19,98 \mathrm{~B}: 0.16$ ).
Other trends may be more surprising. Conditioning on sequence results in a lower structure prediction loss than conditioning on secondary structure (98B; sequence: 3.13 , secondary structure: 3.37). There are some diminishing returns to scale for the prediction of structure, function, SASA, and secondary structure. However, this diminishing is not observed for sequences, where we observe a clear loglinear relationship between pre-training FLOPS and NLL, regardless of conditioning.
To assess the model's unconditional generation capability, we sampled 100 protein lengths randomly from the PDB and generated 1,024 sequences for each using ESM3 98B with a constant temperature of 0.7 . The sampled length distribution is shown in Fig. S13A. Structures for each sequence were predicted using ESM3 7B, and the distribution of pTM
| Model | CASP14 | CASP15 | CAMEO | | :---: | :---: | :---: | :---: | | ESM2 3B | $0.57(0.49-0.64)$ | $0.57(0.48-0.65)$ | $0.75(0.73-0.77)$ | | ESM3 1.4B | $0.56(0.48-0.64)$ | $0.59(0.50-0.66)$ | $0.76(0.74-0.78)$ | | ESM3 7B | $0.62(0.54-0.70)$ | $0.64(0.56-0.73)$ | $0.82(0.80-0.84)$ | | ESM3 98B | $0.66(0.57-0.74)$ | $0.66(0.57-0.75)$ | $0.85(0.83-0.86)$ |
Table S7.Precision @ L results. Measured on CASP14, CASP15 and CAMEO for the ESM3 model family. Intervals represent bootstrapped $95 \%$ confidence intervals.
| | Iterative $/ O\left(L^{3}\right)$ | | | $\operatorname{Argmax} / O\left(L^{2}\right)$ | | | | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | Model | CAMEO | CASP14 | CASP15 | CAMEO | CASP14 | CASP15 | | 1.4B Open | 0.830 | 0.705 | 0.733 | 0.805 | 0.640 | 0.677 | | 1.4B Overtrained | 0.846 | 0.714 | 0.750 | 0.825 | 0.651 | 0.700 | | 1.4B | 0.807 | 0.693 | 0.697 | 0.775 | 0.608 | 0.636 | | 7B | 0.870 | 0.742 | 0.764 | 0.852 | 0.607 | 0.726 | | 98B | 0.895 | 0.763 | 0.801 | 0.884 | 0.719 | 0.770 | | ESMFold | 0.865 | 0.728 | 0.735 | | | | | AlphaFold2 | 0.904 | 0.846 | 0.826 | | | |
Table S8. Protein structure prediction results. We benchmark ESMFold, ESM3 models, and Alphafold2. Left side: ESM3 iterative inference of structure tokens conditioned on sequence. Because iterative inference is $O\left(L^{3}\right)$ in length of a protein sequence, it is comparable to ESMFold and AF2, both of which share the same runtime complexity. Right panel: Single-pass argmax structure token given sequence. In all cases, the more difficult the dataset, the more iterative decoding appears to help - 98B has a +4.4 LDDT boost on CASP14, compared to a +1.0 LDDT boost on CAMEO. Both the Open and Overtrained models are both trained up to 200k steps. The plain 1.4B model is used for scaling comparisons, and is trained to $50 \mathrm{k}$ steps.
| | | Conditioning | | | | | | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | | Model | Sequence | Structure | Function | SASA | Secondary Structure | | | $1.4 \mathrm{~B}$ | 2.31 | 1.71 | 2.28 | 1.81 | 2.02 | | Sequence | $7 \mathrm{~B}$ | 2.04 | 1.43 | 2.00 | 1.47 | 1.74 | | | 98 | 1.84 | 1.21 | 1.76 | 1.21 | 1.50 | | | $1.4 \mathrm{~B}$ | 4.09 | 4.98 | 4.93 | 4.39 | 4.42 | | Structure | $7 \mathrm{~B}$ | 3.42 | 4.2 | 4.18 | 3.62 | 3.71 | | | 98 | 3.13 | 3.85 | 3.8 | 3.24 | 3.37 | | | $1.4 \mathrm{~B}$ | 1.81 | 1.98 | 4.52 | 2.29 | 2.24 | | Function | $7 \mathrm{~B}$ | 1.22 | 1.47 | 3.75 | 1.67 | 1.70 | | | 98 | 0.93 | 1.20 | 3.63 | 1.41 | 1.40 | | | $1.4 \mathrm{~B}$ | 1.78 | 1.81 | 2.42 | 2.48 | 2.10 | | SASA | 7B | 1.57 | 1.66 | 2.26 | 2.31 | 1.92 | | | 98 | 1.46 | 1.56 | 2.15 | 2.23 | 1.82 | | Secondary | $1.4 \mathrm{~B}$ | 0.42 | 0.24 | 0.70 | 0.50 | 0.83 | | Structure | $7 \mathrm{~B}$ | 0.31 | 0.19 | 0.57 | 0.31 | 0.6 | | | 98 | 0.26 | 0.16 | 0.50 | 0.25 | 0.54 |
Table S9. Negative log-likelihood of each track conditioned on other tracks. Each row is a model size, generating a particular modality. Each column is the conditioning. The diagonal, highlighted with italics, are the unconditional NLL of each track. We observe that indeed adding conditioning improves NLL in all cases.
Figure S11. Conditional and unconditional scaling behavior for each track. Loss is shown on CAMEO (Appendix A.3.2
Figure S12. Distribution of $p T M$ and $p L D D T$. Measured on natural (left) and generated (right) sequences under ESM3 7B structure prediction. Generated sequences show a clearly lower correlation (Pearson $\mathrm{r} 0.79 \mathrm{vs}$. 0.85 ) as well as a mode of sequences with high pLDDT but low pTM. Natural sequences are from the test set (Appendix A.3.2), generations are unconditional generations from ESM3 98B. and pLDDT are shown in Fig. S13B. ESM3 generates more high-quality structures than ESM2, which was trained using a simple MLM objective over sequence only with a fixed mask rate. Sequence similarity to the training set was computed using mmseqs2 (73) with the following parameters: --cov-mode 2 -c 0.8 -s 6.0. Proteins generated unconditionally are similar-but not identical-to proteins found in the training set (Fig. S15) and have high coverage of the training set (Fig. 1E), demonstrating that the model has properly fit the training distribution and does not exhibit mode collapse. We observe a cluster of generations with very high sequence identity to the training set; these correspond to antibody sequences, with the framework regions accounting for the high sequence identity.
We use pTM for evaluating structure predictions from ESM3 instead of pLDDT. This is because pLDDT can be miscalibrated for generated structures and can overestimate the confidence of a prediction. pLDDT is biased towards local structural confidence, which can result in pathologies such as very long alpha helices with high pLDDT at all positions. pTM is a more global measure of structural confidence, and is more robust to these pathologies. Fig. S12 shows that $\mathrm{pTM}$ and pLDDT correlation drops for generated sequences $($ Pearson $\mathrm{r}$ : natural $=0.85$, generation $=0.79$ ), and a clear pattern of high pLDDT ( $>0.8$ ) but low pTM $(<0.6)$ emerges.
To visualize the distribution of unconditional generations, we compute sequence embeddings by extracting the final layer outputs produced by running ESM3 7B with sequence inputs only. Protein-level embeddings are computed by averaging over all positions in the sequence to produce a 2560 -dim embedding. We then project these embeddings into two dimensions using a UMAP projection (90) fit on a background distribution of 50,000 randomly sampled sequences from UniProt with minimum distance 0.1 and number of neighbors 25 . Examples are selected by computing structural clusters with Foldseek-cluster (using default parameters) and sampling the example with highest ESM3 pTM from each cluster. A subset of these cluster representatives are shown in Fig. 1E.
To assess whether ESM3 is biased towards particular secondary structures, we use DSSP to predict the three-class secondary structure of the high-confidence ( $\mathrm{pTM}>0.8$, mean $\mathrm{pLDDT}>0.8$ ) generations and measure the percentage of residues that form alpha helices and beta sheets. When compared to a background distribution computed over the PDB, we find that ESM3 closely matches the secondary structure distribution of known proteins (Fig. S13D), unlike other methods which preferentially generate helical structures $(14,23,25)$. Finally, to confirm that the structures predicted with high confidence by ESM3 are designable, we inverse folded and re-folded each using ESM3 7B. The ma- jority of generations successfully re-folded with TM-score of greater than 0.8 to the hallucinated structures, demonstrating that ESM3 has high self-consistency for its own high-confidence designs (Fig. S13C).
To explore alternative ways of generating proteins, we assess the quality of proteins generated by a chain-of-thought $(\mathrm{CoT})$ procedure in which ESM3 7B generates the secondary structure (SS8 tokens), then the 3-D backbone coordinates (structure tokens), followed by the amino acid sequence (sequence tokens) (Fig. S14). We compare the quality of amino acid sequences generated from this CoT procedure with the above method of unconditionally directly generating amino acid sequences. We find that the CoT procedure generates sequences that have higher confidence ESM3predicted structures than the directly-generated sequences as measured by pTM and mean pLDDT (Fig. S14A). Compared to high-confidence ( $\mathrm{pTM}>0.8$, mean $\mathrm{pLDDT}>0.8$ ) directly-generated sequences, the high-confidence subset of CoT-generated sequences are also more designable: the CoT-generated sequences have predicted structures whose inverse folded, then re-refolded structures have higher TMscore to the originally predicted structure (Fig. S14C). The CoT-generated sequences show a small bias towards higher alpha and beta proportion compared to those generated directly (Fig. S14D).
To evaluate ESM's ability to follow prompts, we use a set of held-out proteins as described in Appendix A.3.2. The test set is further filtered to remove proteins with length greater than 1024, which removes 7 proteins from the test set. To construct prompts for the structure coordinate, secondary structure, and SASA tracks, we sample a random span of length $15 \%$ of the original protein length. The model is then shown the corresponding track for the randomly sampled span, and is tasked with generating the sequence for the entire protein. For example, for the structure track, for a protein of length 100 , we may sample a random span of 15 residues from residue $20-35$. The model would then have to generate a protein sequence of length 100 conditioned on structure coordinate conditioning from residues 20-35 derived from the original test protein. This same procedure is applied for the secondary structure and SASA tracks. For the function track, we form the prompt by tokenizing the keywords form the InterProScan annotations associated with each sequence. The ESM3 7B model is used for all generations with a temperature of 0.7 and $L$ decoding steps (where $L$ is the length of the sequence). The model generates 64 sequences per prompt, which we use to compute pass64.
To evaluate the generations, we use ESMFold to fold the sequences generated by ESM3. For the structure coordinate, secondary structure, and SASA tracks, the relevant align-
Figure S13. Unconditional generation of high-quality and diverse proteins using ESM3. (A) Distribution of sequence length in the unconditional generation dataset. (B) Mean pLDDT and pTM of unconditional generations from ESM3 compared to sequences designed using the 3B-parameter ESM2 model. (C) Round-trip success rate of high-confidence generations using ESM3. Predicted structures were inverse folded to predict a new sequence and then re-folded to produce a new structure. Success was measured by a TM-score of greater than 0.8 between the original and refolded designs. (D) Secondary structure composition of unconditional generations relative to the distribution of proteins in the PDB, which is shown in gray.
Figure S14. Generation of sequences using chain-of-thought. SS8 tokens are generated first, followed by structure tokens, then amino acid sequence with the ESM3 7B model. (A) Distribution of mean pLDDT and pTM of sequences generated by chain-of-thought ("ss8 first") compared to directly generating the sequence ("sequence only"). (B) Sample generations of SS8 tokens and the predicted structure of its corresponding CoT sequence. (C) TM-score between predicted structures of high-confidence ( $\mathrm{pTM}>0.8$, mean pLDDT $>0.8$ ) generated sequences and their corresponding inverse folded, then re-folded structures. (D) Comparison of the secondary structure composition of high-confidence generated sequences to the distribution of proteins in the PDB. ment metrics (backbone cRMSD, 3-class secondary structure accuracy, and SASA Spearman $\rho$ ) can be calculated on the relevant span in the ESMFold-predicted structure and the original template protein. Continuing the previous example for the structure track, we would compute the RMSD between residues 20-35 in the ESMFold structure predicted of the ESM3-generated sequence and residues 20-35 of the original test protein. For the function annotation track, we run InterProScan (38) on each generated sequence and extract function keywords from the emitted annotations. We report function keyword recovery at the protein level, computing the proportion of all function keywords in the prompt which appear anywhere in the function keywords from the InterProScan annotations of the generation.
To test the ability of ESM3 to generalize beyond its training distribution under prompting, we evaluate two prompting scenarios. First, we identify proteins which were deposited in the PDB after our training cutoff (December 2020) and choose eight with $\mathrm{TM}<0.7$ to any structure in our training dataset (PDB IDs: $2 \mathrm{JVN}$ chain A, $2 \mathrm{KAF}$ chain A, $2 \mathrm{~L} 8 \mathrm{~K}$ chain $\mathrm{A}, 2 \mathrm{MJM}$ chain $\mathrm{A}, 7 \mathrm{ZUO}$ chain $\mathrm{A}, 8 \mathrm{EXF}$ chain B). Using DSSP, we compute the residue-level SS8 and SASA for each of these proteins to prompt ESM3, masking all other tracks. We show in Fig. S15A that the generated proteins are diverse, globular, and closely follow the SS8 and SASA prompts while having no close sequence or structure neighbors in the training set. Interestingly, these proteins are not folded with high confidence or accuracy by ESMFold (mean pTM 0.44 , mean TM-score to reference 0.33), suggesting that these are challenging proteins to fold. The ESM3generated sequences have a similar confidence (mean pTM 0.45 ) but much higher accuracy (mean TM-score 0.64).
Second, we classify the residue-level secondary structure for a set of eight symmetric protein backbones using DSSP. These proteins were previously designed using ESMFold $(5,91)$ and have varying secondary structure (alpha and beta) and varying symmetries (5-fold and 8 -fold). Again, ESM3 is able to design these proteins successfully with high confidence ( $\mathrm{pTM}>0.8$, pLDDT $>0.8$ ) and low sequence similarity to the training set Fig. S15B. The structural similarity is moderate for these designs due to the high structural conservation of the protomer units in each design. All designs are generated using a constant temperature of 0.7 with $\mathrm{L} / 2$ decoding steps, where $\mathrm{L}$ is the protein length. We sample 256 sequences for each prompt and filter generations by pTM ( $>0.8$ ), pLDDT ( $>0.8$ ), and accuracy in satisfying the SS8 prompts ( $>0.8$ ). Final examples were selected from these filtered designs by visual inspection. Sequence similarity to the training set was computed using the same procedure as the unconditional generations, and structure similarity was computed using Foldseek (39) in TM-score mode (alignment-type 1) with sensitivity -s 7.5.
ESM3 is able to compose multimodal prompts across its input tracks-sequence, structure, SS8, SASA, and function keywords-to generate proteins with novel characteristics. To demonstrate this, we augment the standard functional motif scaffolding task (i.e., partial structure and sequence prompts) with additional conditioning to specify the type of scaffold for ESM3 to design. The functional sites comprise a combination of ligand binding sites coordinated by residues remote in sequence and those defined by short local motifs. For each motif, the coordinates and amino acid identities of all residues from the reference PDB structures are input to the model, with random shuffling and augmentation of the gaps between each active site. See Appendix A.4.5 for a description of this augmentation procedure and the specifications of the ligand-binding sites chosen. In addition to these sites, we also create a set of 12 partial sequence and structure prompts derived from conserved functional motifs (Table S10). These motifs are defined using a combination of the benchmark dataset in Watson et al. (23) and conserved sequence patterns from the Prosite database (92).
The scaffold conditioning is defined using either SS8 tokens (to specify secondary structure composition) or function keywords defined by InterPro accession numbers (to specify a particular fold). For each combination of functional site and scaffold prompt, we sample between 256 and 2048 times to generate proteins with diverse and novel characteristics. All designs were generated with the 7B-parameter model, a constant temperature of 0.7 , and $L / 2$ decoding steps for a protein of length $L$.
Secondary structure prompting. We generated proteins under four main classes of secondary structure composition: mostly alpha helices, mostly beta sheets, and mixed alphabeta proteins (split into alpha/beta, alpha/beta/alpha, and beta/alpha/beta topologies). For each generation, we prompt the model with a random set of SS8 spans up to a total length $L$, with mask tokens in between. For example, an all-alpha SS8 prompt for a protein of length $L=20$ might look like __HHHH $\mathrm{HHHHH}$ $\mathrm{HH}$ and a beta-alpha-beta prompt might look like _EEEHHHHHEE_, where H is a residue within an alpha helix and $\mathrm{E}$ is a residue in a beta strand. We then combine this with the augmented partial structure and sequence tracks given by a functional site motif. To increase the diversity of the scaffolds and maximize the probability of generating physically realizable prompt combinations, we generate between 256 and 1024 designs for each combination of SS8 and functional site motif. For each generation, we uniformly sample a random length $L$ between 150 and 400 . Then, we produce a set of secondary structure spans with length 5-20 residues, each separated
A
B
Figure S15. Prompting ESM3 to generalize beyond its training distribution. (A) Proteins designed using SS8 and SASA prompts derived from recent structures in the PDB with low structural similarity to the training set. Prompts along the protein length are visualized above each generation; secondary structure is shown using three-class (alpha $=$ blue, beta $=$ orange, coil $=$ gray) and SASA is shown as a line plot colored by residue index to match the cartoon below. (B) Symmetric proteins designed using SS8 prompting. Histograms show the similarity to the nearest training set protein by structure (TM-score) and sequence (sequence identity) compared to unconditional generation.
| Motif | PDB ID | Chain ID | PDB Residue Identifiers | | ---: | :---: | :---: | :---: | | ACE2 binding | $6 \mathrm{vw} 1$ | $\mathrm{~A}$ | $19-89,319-366$ | | Ferredoxin | $6 \mathrm{6} 6 \mathrm{r}$ | $\mathrm{A}$ | $1-44$ | | Barstar binding | $7 \mathrm{mrx}$ | $\mathrm{B}$ | $25-47$ | | P53 binding | $1 \mathrm{ycr}$ | $\mathrm{B}$ | $19-28$ | | PD-1 binding | $5 \mathrm{ius}$ | $\mathrm{A}$ | $63-83,119-141$ | | DNA-binding helix-turn-helix | $11 \mathrm{cc}$ | $\mathrm{A}$ | $1-52$ | | P-loop | $5 \mathrm{ze} 9$ | $\mathrm{~A}$ | $229-243$ | | Double EF-hand | $1 \mathrm{a} 2 \mathrm{x}$ | $\mathrm{A}$ | $103-115,139-152$ | | Lactate dehydrogenase | $11 \mathrm{db}$ | $\mathrm{A}$ | $186-206$ | | Renal dipeptidase | $1 \mathrm{itu}$ | $\mathrm{A}$ | $124-147$ | | Ubiquitin-activating enzyme E1C binding | $1 \mathrm{yov}$ | $\mathrm{B}$ | $213-223$ | | DNA topoisomerase | $1 \mathrm{a} 41$ | $\mathrm{~A}$ | $248-280$ |
Table S10. Functional motif definitions for conserved regions. by a gap of 3-10 residues, such that the total length adds up to $L$. Finally, to avoid incompatibility between the partial structure and secondary structure constraints, we also mask the SS8 tokens at positions where structure is specified by the functional site prompt. Secondary structure-prompted designs was assessed by running DSSP on the designed sequence and measuring the fraction of prompted residues which were assigned the correct secondary structure. Success was determined by a pTM $>0.8$, all-atom cRMSD $<$ 1.5 for the functional site, and SS8 accuracy $>0.8$.
Keyword prompting. To prompt the model to generate proteins with a specific fold, we extracted the set of InterPro tags associated with a set of proteins from the CAMEO test set for which ESM3 achieved keyword recovery of greater than $80 \%$ (Fig. 2A). These tags were then converted into keywords and used to prompt the model in combination with the partial sequence and structure constraints. The list of prompts and function tags is given in Table S11. Keywordprompted designs were assessed using a self-consistency evaluation, i.e. whether the model successfully predicts any of the prompted InterPro accessions for the designed sequence. Success was determined by a pTM $>0.8$, all-atom $c$ RMSD $<2.0$, and number of InterPro accessions recovered $>0$.
We assess novelty of each motif-scaffold combinations by measuring the TM-score between the generated scaffold and the chain from which the motif is derived (Table S12). This confirms that the model is not retrieving the original motif scaffold, particularly for secondary structure-prompted scaffolds where we do not provide any explicit instructions to produce diverse designs. For the motifs derived from ligand binding residues (magnesium, serotonin, calcium, zinc, protease inhibitor 017, and Mcl-1 inhibitor YLT), we additionally use Foldseek to search the PDB for any other proteins which share that motif (as defined by BioLiP (93)), as a more stringent evaluation of novelty. For all but zincbinding and magnesium-binding motifs, Foldseek finds no significant hits at an E-value threshold of 1.0. The hits discovered for zinc and magnesium have only modest TMscore ( 0.76 and 0.64 ), demonstrating that the model still finds novel scaffolding solutions for these ligands. To assess whether the generated scaffolds are likely to be designable, we measure a self-consistency TM-score under orthogonal computational models by inverse-folding the designed structure with ESM-IF (94) (using a temperature of 0.5 ) and re-folding with ESMFold (5). We report the best scTM over 8 inverse folding designs in Table S12.
First, we describe the procedure for generating the protein compression example shown in Fig. 2D. A series of prompts of length 150 were constructed. The sequence and struc- ture of the catalytic triad of trypsin (PDB 1Y3V) (H57, D102, S195) were placed in the prompt using the following procedure: three random residue numbers between 20 and 130 were sampled such that the minimum pairwise difference in position between each of the residues was no less than 20. Then, H57 from the template trypsin was placed at the lowest sampled number, D102 at the second lowest, and S195 at the largest number, thus respecting the left-to-right ordering of the catalytic triad in the template trypsin. 128 prompts were generated by this procedure. Each of these prompts was combined with a function keyword prompt derived from the template protein, specifically InterPro (38) tags IPR001254 (serine proteases, trypsin domain) and IPR009003 (peptidase S1, PA clan), to arrive at a final set of 128 prompts. The base ESM 7B model was then prompted to generate the sequence of the remaining 147 residues of the protein conditioned on the randomly placed catalytic triad sequence and structure coordinates and function keywords. $L=150$ decoding steps were used with a temperature of 0.7 , with 32 generations per prompt. Generations were then filtered by active site cRMSD, ESM3 pTM, and InterPro Scan keyword outputs, with the generation shown in Fig. 2D selected finally by visual inspection.
Generation quality was measured using ESMFold (5) pTM of the generated sequence, in addition to self-consistency. For self-consistency, we inverse fold the ESM3-predicted structure of the generation with ESM-IF1 (94) 8 times and re-fold with ESMFold, reporting the mean and std of the TM-scores between the 8 ESMFold-predicted structures and the ESM3-predicted structure. To perform a blast search of the sequence, we use a standard Protein Blast search (51). We set the max target sequences parameter to 5000 and sort results by sequence length and sequence identity, selecting the first sequence that is a serine protease. This yields the reference WP_260327207 which is 164 residues long and shares $33 \%$ sequence identity with the generation.
We showcase two further examples of protein editing. First, ESM3 is prompted to bury an exposed helix in a protein with an alternating alpha-beta sandwich fold. The prompt is constructed as follows: the prompt is of the same length as the template protein (PDB 1LBS). We identify a buried helix (mean SASA $0.32 \AA^{2}$ ) between residues 106-116 of the template protein. Structure coordinates from this region are placed in the prompt at the same residue indices, to prompt ESM3 to generate the same helix. This is composed with a SASA prompt of 40.0 for each of the 11 helix residues, prompting ESM3 to place this helix on the surface of the protein. Finally, we prompt with the secondary structure of 5 central beta strands surrounding the buried helix, residues 33-36, 62-65, 99-103, 125-130, and 179-182. ESM3 7B is then used to generate 512 protein sequences conditioned on this prompt using $\frac{L}{2}$ decoding steps and a temperature of 0.7. Designs are filtered by ESM3 pTM and adherence
| Scaffold | Reference | InterPro tags | Total Length |
| :---: | :---: | :---: | :---: |
| Beta propeller | $8 \sin \mathrm{A}$ | IPR001680 (1-350)
IPR036322 (1-350)
IPR015943 (1-350) | 353 |
| TIM barrel | $7 \mathrm{rpnA}$ | IPR000652 (0-248)
IPR020861 (164-175)
IPR035990 (0-249)
IPR013785 (0-251)
IPR000652 (2-249)
IPR022896 (1-249) | 252 |
| MFS transporter | 4ikvA | IPR011701 (1-380)
IPR020846 (1-380)
IPR036259 (1-380) | 380 |
| Immunoglobulin | $7 \mathrm{sbdH}$ | IPR036179 (0-116; 124-199)
IPR013783 (0-206)
IPR003597 (124-202)
IPR007110 (0-115; 121-207)
IPR003599 (6-115)
IPR013106 (11-114) | 209 |
| Histidine kinase | 8dvqA | IPR003594 (47-156)
IPR003594 (47-158)
IPR004358 (118-137)
IPR004358 (141-155)
IPR004358 (101-112)
IPR005467 (0-158)
IPR036890 (4-159)
IPR036890 (3-156) | 166 |
| Alpha/beta hydrolase | 7yiiA | IPR029058 (0-274)
IPR000073 (26-265) | 276 |
Table S11. InterPro tags extracted from CAMEO test set proteins for prompting with fold specification.
| | | | | | ---: | ---: | :---: | :---: | | Site | Scaffold | Novelty (TM to original) | Designability (scTM) | | 017 | beta | 0.264 | 0.967 | | ACE2 | alpha | 0.606 | 0.871 | | CA | Immunoglobulin | 0.441 | 0.781 | | MG | ab-hydrolase | 0.293 | 0.969 | | TIM-barrel | 0.328 | 0.980 | | | Renal-dipeptidase | alpha-beta-alpha | 0.644 | 0.933 | | SRO | mfs-transporter | 0.345 | 0.992 | | Topoisomerase | histidine-kinase | 0.269 | 0.948 | | YLT | alpha-beta | 0.229 | 0.899 | | ZN | alpha | 0.567 | 0.996 |
Table S12. Novelty and designability metrics. Metrics are shown for motif scaffolds shown in Fig. 2C. Novelty is measured by computing the TM-score to the original scaffold from which the motif is derived. Designability is measured by self-consistency TM-score over eight samples by inverse folding with ESM-IF and refolding with ESMFold. All designs are distinct from their original scaffolds while retaining high designability. to the SASA prompt. The final generation is chosen by visual inspection. The generation is evaluated as described above (ESMFold pTM 0.71, scTM mean 0.82, std 0.045). Examining the generation, ESM3 is able to satisfy the input constraints: the generated protein maintains the structure of the helix (cRMSD $0.18 \AA$ ) and the alternating alpha-beta fold (both the generation and the template have 7 strands alternating with helices), while exposing the helix motif to the surface (mean SASA $28.35 \AA^{2}$ ). Furthermore, the generation is structurally distinct: a Foldseek search (39) of AlphaFold-DB, ESMAtlas, and PDB in TM-align mode reveals no hit with TM-score greater than .76.
We also use ESM3 to generate an idealized TIM Barrel with 11-fold symmetry. This generation is undertaken in two steps. First, we derive a secondary structure and function keyword prompt from a reference TIM Barrel (PDB 5EKY). The secondary structure of the reference protein is computed using DSSP and then idealized to construct a prompt for ESM3. To construct the secondary structure prompt, the length of each helix and strand is fixed at 7 residues. Each helix and strand region is then separated by 3 mask tokens, with a mask token appended to the $\mathrm{N}$ and $\mathrm{C}$ termini of the prompt as well. This yields a secondary structure prompt of total length 159 , which is combined with a function keyword prompt derived from the reference protein: keywords are derived from IPR013785 (aldolase-type TIM barrel) and IPR000887 (KDPG/KHG aldolase). ESM3 7B is then used to generate 256 samples with $L$ decoding steps and a temperature of 0.7 . The design shown is chosen by filtering by ESM3 pTM and visual inspection. In the second step, the secondary structure prompt from the first step is expanded to contain 11 helix-strand subunits, for a total prompt length of 225 residues (4 mask tokens are now appended to the $\mathrm{N}$ and $\mathrm{C}$ termini, rather than just 1). ESM3 7B is then used to generate 256 samples with $L$ decoding steps and a temperature of 0.7 , with generations filtered by ESM3 pTM and visual inspection. The generation is evaluated as described above (ESMFold pTM 0.69, scTM mean 0.97, std 0.011). The generation is structurally distinct: a Foldseek search (39) of AlphaFold-DB, ESMAtlas, and PDB in TM-align mode reveals no hit with TM-score greater than . 61 .
Please summarize for an expert
AI: The document details the evaluation of ESM3, a large-scale language model designed for protein structure and function prediction, highlighting its generative capabilities and ability to learn representations for predictive tasks. Key aspects include:
Model Descriptions: ESM3 models are described in varying scales - from 1.4B to 98B parameters, trained on a substantial amount of token data, indicating a scalable architecture aimed at accommodating extensive protein sequence and structure data.
Data and Benchmarking: Benchmarking involves assessing the models on a held-out test set of 902 proteins, along with evaluations on significant protein structure prediction benchmarks such as CASP14 and CASP15. This ensures rigorous testing against known and challenging datasets.
Representation Learning and Contact Prediction: ESM3 applies LoRA for finetuning, a less memory-intensive method than full weight finetuning, showing substantial effectiveness in contact prediction across various test sets. This demonstrates ESM3's proficiency in learning meaningful representations that can accurately predict protein contacts.
Structure Prediction: ESM3's ability to predict protein structures directly, without further finetuning, was tested through different decoding strategies. Results showed its competitive edge, particularly when employing iterative decoding, indicating a significant capability in structural prediction tasks, rivaling that of advanced models like AlphaFold2.
Generative Capabilities and Conditional Likelihood: The model displayed impressive generative abilities, with evaluations highlighting the conditional likelihood performance across different tracks, underscoring ESM3's robustness as a generative model.
Unconditional Generation and Diversity: ESM3 demonstrated its capacity to generate high-quality and diverse protein sequences and structures unconditionally, exhibiting a high degree of novelty and biological relevance without overfitting to the training set, which was further validated through successful protein design experiments.
Design and Prompting Evaluations: Detailed experiments showed ESM3's flexibility in accommodating complex prompts, enabling the creation of proteins with specified functions, structures, or properties—a pivotal capability for explorative protein engineering and design.
Composing Prompts and Novelty in Design: The document highlighted ESM3's ability to innovate by generating proteins based on compositional prompts, blending different elements like functional motifs with various structural scaffolds, showcasing the model's utility in creating novel proteins with desired characteristics.
These findings illustrate ESM3's comprehensive approach to modeling protein sequences and structures, encompassing predictive accuracy, generative capabilities, and design novelty. Its ability to handle complex prompts, along with its performance across various evaluations, positions ESM3 as a powerful tool in computational biology, protein research, and biotechnological innovation.
Since the introduction of RLHF (40) there have been a number of algorithms developed to tune large models trained via unsupervised learning to better follow instructions and generally align their generations to user preferences (41, 42, 95, 96). We use IRPO (Iterative Reasoning Preference Optimization) due to its simplicity in implementation and good performance. The IRPO loss combines supervised finetuning with contrastive learning from preference pairs. IRPO operates on a dataset $\mathcal{D} \sim\left(y{w}, y{l}, x\right)$ consisting of prompt $x$ and a pair of completions $y{w}$ (preferred) and $y{l}$ (not preferred). It also operates on two separate models: the reference model $\pi{\text {ref }}$ and the current model $\pi{\theta}$. The reference model $\pi{\text {ref }}$ is the fixed base model of the same scale, and the current model $\pi{\theta}$ is the model being optimized.
$$ \begin{align} \mathcal{L}{\mathrm{IRPO}}\left(\pi{\theta} ;\right. & \left.\pi{\mathrm{ref}}\right)=\mathcal{L}{\mathrm{NLL}}+\alpha \mathcal{L}{\mathrm{DPO}}= \ & -\mathbb{E}{\left(x, y{w}, y{l}\right) \sim \mathcal{D}}\left[\frac{\log \pi{\theta}\left(y{w} \mid x\right)}{\left|y{w}\right|+|x|}+\right. \ \alpha \log \sigma & \left.\left(\beta \log \frac{\pi{\theta}\left(y{w} \mid x\right)}{\pi{\mathrm{ref}}\left(y{w} \mid x\right)}-\beta \log \frac{\pi{\theta}\left(y{l} \mid x\right)}{\pi{\mathrm{ref}}\left(y_{l} \mid x\right)}\right)\right] \tag{2} \end{align} $$
The IRPO loss contains two terms. The $\mathcal{L}{\text {NLL }}$ term maximizes the $\log$ likelihood of the preferred example normalized by the length of the sequence, providing signal to reinforce the good generations from the model. The $\mathcal{L}{\text {DPO }}$ term is the contrastive preference tuning term, which increases the difference in log likelihoods between the preferred and not preferred examples while staying close to the reference model (41). The use of the reference model serves as a regularizer to prevent overfitting to the preference dataset, which can often be small. There are two hyperparameters, $\alpha$ and $\beta$. $\alpha$ weights the relative importance of the supervised with the preference loss and the $\beta$ parameter controls how close we stay to the reference model: the higher the beta, the closer we stay. We minimize this loss with respect to the current model parameters $\theta$.
ESM3 is a multi-modal model so the prompt can be any combination of the input tracks of (partial) sequence, structure, and function and the generation y can be any of the output tracks. In our experiments we always generate the amino-acid sequence so this will be our running example from now on. Since an amino-acid sequence $y$ can be generated from prompt $x$ in many multi-step ways computing the full likelihood $\pi(y \mid x)$ would involve integrating over all possible multi-step decoding paths. Since this is intractable, we use a surrogate that mirrors pre-training, shown in Eq. (3) and described below.
$$ \begin{equation} \log \pi(y \mid x) \approx \mathbb{E}{m}\left[\sum{i \in m} \log p\left(y{i} \mid y{\backslash m}, x\right)\right] \tag{3} \end{equation} $$
To approximate the likelihood of a generation $y$ from prompt $x$, we mask $y$ with a mask sampled from a linear noise schedule, prompt ESM3 with $\left{y_{\backslash m}, x\right}$, and compute the cross-entropy of ESM3 logits with the masked positions of $y$. During training, the same mask is used to compute the likelihoods for the reference policy vs current policy, as well as for the preferred sample vs non preferred sample.
Figure S16. Multimodal protein editing with ESM3. (A) ESM3 exposes a buried helix in an protein while maintaining the alternating alpha-beta sandwich fold of the protein. (B) ESM3 is used in a two-step iterative edit, where first secondary structure prompting and function prompting are used to idealize a reference TIM barrel. Secondary structure prompting is then used to increase the number of subunits in the TIM barrel from 8 to 11 .
Rearranging the DPO term of the loss function gives some insight into how it finetunes the model for the preference pairs.
$$ \begin{align} \mathcal{L}{\mathrm{DPO}}\left(\pi{\theta} ;\right. & \left.\pi{\mathrm{ref}}\right)= \ & \mathbb{E}{\left(x, y{w}, y{l}\right) \sim \mathcal{D}}\left[-\log \sigma\left(-\beta z{\theta}\left(x, y{l}, y_{w}\right)\right)\right] \tag{4} \end{align} $$
where
$$ \begin{aligned} z{\theta}\left(x, y{l}, y{w}\right) & =\log \frac{\pi{\theta}\left(y{l} \mid x\right)}{\pi{\mathrm{ref}}\left(y{l} \mid x\right)}-\log \frac{\pi{\theta}\left(y{w} \mid x\right)}{\pi{\mathrm{ref}}\left(y{w} \mid x\right)} \ & =\log \frac{\pi{\mathrm{ref}}\left(y{w} \mid x\right)}{\pi{\mathrm{ref}}\left(y{l} \mid x\right)}-\log \frac{\pi{\theta}\left(y{w} \mid x\right)}{\pi{\theta}\left(y_{l} \mid x\right)} \end{aligned} $$
The function $f(z)=-\log \sigma(-\beta z)=\log (1+\exp (\beta z))$ is the softplus function, and is an approximation of the hinge function; in other words $f(z)=\beta z$ when $z>>0$ and $f(z)=0$ when $z \ll 0$. Because of this property, there are two cases. In the case where
$$ \begin{equation} \log \frac{\pi{\mathrm{ref}}\left(y{w} \mid x\right)}{\pi{\mathrm{ref}}\left(y{l} \mid x\right)}>>\log \frac{\pi{\theta}\left(y{w} \mid x\right)}{\pi{\theta}\left(y{l} \mid x\right)} \tag{5} \end{equation} $$
$f(z)$ is in the linear regime, so the loss function is simply maximizing the likelihood ratio $\log \frac{\pi{\theta}\left(y{w} \mid x\right)}{\pi{\theta}\left(y{l} \mid x\right)}$. In the case where
$$ \begin{equation} \log \frac{\pi{\text {ref }}\left(y{w} \mid x\right)}{\pi{\text {ref }}\left(y{l} \mid x\right)} \ll \log \frac{\pi{\theta}\left(y{w} \mid x\right)}{\pi{\theta}\left(y{l} \mid x\right)} \tag{6} \end{equation} $$
the loss has saturated. This ensures that we do not deviate too far from the reference model.
These dynamics also hold true in the case of ESM3 finetuning. Although we use a surrogate instead of the true likelihood, the loss will increase the surrogate of the preferred pair over the non preferred pair until the current model deviates too much from the reference model.
Possibly the most important part of preference tuning is to decide how to bucket generations into preferences. The desired objectives for a generation are quality and correctness. Quality refers to the viability of the sequence to be a stable protein. Correctness refers to the extent to which it follows the given prompt; also called prompt consistency. This section only deals with structure coordinate prompts, so prompt consistency can be measured via constrained site RMSD (cRMSD), which is the RMSD between the prompt coordinates and the corresponding coordinates in the predicted structure of the generated sequence. Sequence quality can be measured via predicted-TM (pTM) of a structure predictor on the generated sequence.
As with any metric, especially one which is really a surrogate such as a structure predictor, there is a risk of over optimizing: the model keeps improving the specific metric e.g. in our case pTM but the actual property of interest, the viability of the sequence to be a stable protein, stops correlating with the metric (97). Using orthogonal models to rank our training dataset vs to perform evaluation helps mitigate this.
To create the training datasets, generations are evaluated according to cRMSD and pTM of ESM3 7B to maintain a consistent structure predictor across all datasets. After the preference tuning phase, the generations from the tuned models are evaluated with ESMFold cRMSD and pTM as an orthogonal model. Training on ESM3 derived metrics while evaluating on ESMFold derived metrics should reduce the risk of over optimization for adversarial generations.
All ESM3 model scales are trained with the IRPO loss (Eq. (2)) on their respective preconstructed training datasets consisting of structure coordinate prompts and generations of various difficulty. The datasets have 16 generations each for 30,000 prompts from the respective ESM3 model. Preference selection is determined via a threshold of metrics. A sample is considered "good" if it has ESM3 7B pTM $>0.8$ and backbone cRMSD to its structure prompt $<1.5 \AA$.
Each "good" sample is paired with a "bad" sample to create a preference pair. We found that enforcing a gap between metrics of paired generations improves results, so to qualify as a "bad" sample the generation must have a delta $\mathrm{pTM}=\mathrm{pTM}{\text {good }}-\mathrm{pTM}{\text {bad }}>=0.2$ and delta backbone $c R M S D=c R M S D{\text {good }}-c^{2} M S D{\text {bad }}<-2 \AA$. Each prompt can have multiple preference pairs, and prompts with no valid preference pair are discarded.
The structure prompts are composed of a variety of proteins adapted from our pre-training pipeline. $50 \%$ of the prompts are synthetic active sites, while the other $50 \%$ are structure coordinates randomly masked with a noise schedule. All of the structure prompts are derived from PDB structures with a temporal cutoff of before May 1st, 2020.
The synthetic active sites are derived by finding sequences from PDB with coordinating residues. For these structures, the amino acid identities are included in the prompt.
The remaining structure track prompts are masked according to a cosine noise schedule. $50 \%$ of the noise scheduled prompts are masked in completely random positions, and the other $50 \%$ are masked according to an autocorrelation mechanism that prefers sequentially masked positions.
Each model's training dataset consists of generations of its own reference model. For each prompt, we generate samples from the corresponding ESM3 model scale using iterative decoding with $L / 4$ steps, where $L$ is the length of the prompt. We anneal the temperature from 1.0 to 0.5 over the decoding steps.
Atomic coordination tasks require the generation of proteins which satisfy challenging tertiary interaction constraints. The model is prompted with the sequence and coordinates of a set of residues which are near in 3D space, but distant in sequence. To evaluate performance on these tasks, we curate a dataset of 46 proteins with ligand binding sites from the Biolip dataset (93). All selected proteins were deposited in the PDB after the training set cutoff date (2020-12-01). The coordinating residues shown to the model are given by the ligand binding sites defined in the Biolip dataset (Table S13).
ESM3 is prompted with the sequence and coordinates of the residues for a particular ligand binding site. We ask ESM3 to generate novel structures by applying multiple transformations to the prompt. The total sequence length is sampled evenly to be 150,250 , or 350 residues (regardless of the original sequence length). Next, we define a contiguous span of coordinating residues to be prompt residues with fewer than 5 sequence positions between them. The order and the distance between contiguous spans of residues is shuffled. Together, this ensures that, for example, the original protein will no longer satisfy the prompt. We consider a generation a success if backbone cRMSD $<1.5 \AA$ and $\mathrm{pTM}>0.8$.
We construct a total of 1024 prompts for each ligand and generate a completion for each prompt with the model we are evaluating. We report Pass@ 128, which is an estimate for the fraction of ligands with at least one successful completion after 128 prompts per ligand. We estimate this using an unbiased estimator (Chen et al. (98), Page 3) using the success rate over 1024 prompts. We visualize randomly selected successful generations for both the base model and finetuned model in Fig. S18
To judge the value of preference tuning, we also train a supervised finetuning (SFT) baseline where we finetune the model to increase likelihood of the high quality samples without the preference tuning loss. The 1.4B, 7B, and 98B models solve $14.2 \%, 33.7 \%$, and $44.6 \%$ of atomic coordination tasks at 128 generations, respectively, which improves upon the base models but is much lower than their corresponding preference tuned versions.
Each IRPO model is trained for 1000 steps using RMSProp. The learning rates are $1 \mathrm{e}-5,1 \mathrm{e}-5$, and $5 \mathrm{e}-6$ for the $1.4 \mathrm{~B}$, $7 \mathrm{~B}$, and 98B, respectively, annealed using a cosine schedule after a 150 step warmup. Gradient norms are clipped to 1.0.
For all IRPO runs $\beta=0.05$ and $\alpha=0.8$. The SFT baseline uses the same hyperparameters, but with $\alpha=0.0$ to disregard the preference tuning term.
Please summarize for an expert
AI: The section describes the alignment process for fine-tuning large language models, particularly focusing on the Iterative Reasoning Preference Optimization (IRPO) algorithm. IRPO enhances model performance by integrating supervised fine-tuning with contrastive learning from preference pairs. This process operates on a dataset of preference pairs (preferred and not preferred completions) given a prompt, using both a fixed reference model and a current model under optimization. The IRPO loss comprises two terms: the NLL (Negative Log Likelihood) term aimed at maximizing the log likelihood of the preferred example, and the DPO (Difference in Preference Optimization) term, which contrasts the log likelihoods of preferred and not preferred examples while maintaining closeness to the reference model.
ESM3, a multi-modal model, is used as an example, highlighting its capability to generate amino-acid sequences based on various input prompts. The approximation of full likelihood via a surrogate method is explained, facilitating practical implementation.
The document further delves into the intuition behind preference tuning, illustrating how the algorithm fine-tunes models based on preference pairs through rearrangements of the DPO term. It emphasizes the algorithm’s use of the softplus function to balance the emphasis on preference pairs and prevent excessive deviation from the reference model.
Evaluation metrics, training dataset creation, and the performance of models on atomic coordination tasks are discussed, providing insights into how models are developed, fine-tuned, and assessed for generating protein structures. The preference tuning's effectiveness in improving predictions' quality and correctness is highlighted alongside concerns regarding overoptimization.
Finally, supervised fine-tuning (SFT) is presented as a baseline comparison, revealing that while SFT improves base model performance, preference-tuned models demonstrate superior efficacy on complex tasks. Training hyperparameters for both IRPO and SFT approaches are detailed, offering a comprehensive overview of the methodological framework and parameters critical to optimizing model performance.
ESM3 generates a dim distant GFP B8 and a bright distant protein esmGFP. Details are provided below on com-
| PDB ID | Coordinating Residues | Ligand ID | | :---: | :---: | :---: | | $7 \mathrm{map}$ | D25 G27 A28 D29 D30 G48 G49 V50 | 017 | | $7 n 3 \mathrm{u}$ | I305 F310 V313 A326 K328 N376 C379 G382 D386 F433 | $05 \mathrm{~J}$ | | 7 exd | D103 I104 C107 T108 I174 H176 T182 W306 F309 E313 Y337 | $05 \mathrm{X}$ | | $8 g x p$ | W317 C320 A321 H323 V376 F377 L396 I400 H479 Y502 | $06 \mathrm{~L}$ | | $7 \mathrm{n} 4 \mathrm{z}$ | M66 C67 R124 L130 C134 Y135 D152 F155 | $08 \mathrm{~N}$ | | $7 \mathrm{vrd}$ | A40 S41 H161 Q169 E170 E213 D248 D324 K349 H377 R378 S379 K400 | $2 \mathrm{PG}$ | | $7 \mathrm{zyk}$ | V53 V66 V116 H160 N161 I174 D175 | ADP | | $6 \mathrm{yj} 7$ | K23 V24 A25 Y45 T46 A47 F115 I128 | AMP | | $8 \mathrm{ppb}$ | H185 F198 K209 Q249 D250 L251 D262 K336 I415 D416 | ATP | | $7 \mathrm{knv}$ | E33 F94 E95 D125 | $\mathrm{CA}$ | | 7 xer | Y466 L505 T525 | CLR | | $7 \mathrm{tj} 6$ | F366 G367 T378 R418 | CMP | | $6 x m 7$ | $\mathrm{H} 167 \mathrm{H} 218 \mathrm{H} 284 \mathrm{H} 476$ | $\mathrm{CO}$ | | $7 \mathrm{bfr}$ | Q62 X126 H248 | $\mathrm{CO} 3$ | | $6 x \operatorname{lr}$ | X272 Y495 H496 H581 | $\mathrm{CU}$ | | 6 tnh | N40 A41 S127 T128 Q187 L191 C201 T202 V236 | DGP | | $7 \mathrm{ndr}$ | F73 S101 F102 D103 R106 | EDO | | $8 \mathrm{axy}$ | H68 H109 E144 | $\mathrm{FE}$ | | $7 \mathrm{o6c}$ | E62 E107 Q141 | FE2 | | 8aul | P31 M32 T33 Q106 H185 R237 S319 G320 G321 G342 R343 F369 Y370 | $\mathrm{FMN}$ | | $7 \mathrm{vcp}$ | N37 D38 Q54 F97 S98 R159 D160 E214 Y276 W297 | FRU | | $7 b 7 f$ | G167 T168 G189 W195 | FUC | | $8 \mathrm{~d} 0 \mathrm{w}$ | F73 L136 E137 F329 | GAL | | 7yua | T13 T14 I15 D40 H85 S86 D87 D110 N290 | GDP | | $7 \mathrm{w} 1 \mathrm{a}$ | L44 Y88 L91 I212 | GMP | | $71 j n$ | G71 S72 D91 K236 S253 V254 D309 R310 | GTP | | $6 s 4 \mathrm{f}$ | Y84 N87 K88 V131 Q132 L133 D155 F157 I276 P309 G310 G313 P314 V317 | $\mathrm{KUN}$ | | $7 \mathrm{mg} 7$ | Y12 G98 L99 Y100 A207 D208 G227 R228 | MAN | | 7qow | D12 T118 E268 | $\mathrm{MG}$ | | $7 \mathrm{dmm}$ | E181 E217 D245 D287 | $\mathrm{MN}$ | | $7 \mathrm{qoz}$ | G11 G12 I13 Y34 D35 V36 A86 G87 V126 T127 N128 H185 M235 | NAD | | $7 v 2 r$ | G89 F93 K98 F101 E121 Y204 E209 F229 | $\mathrm{NAI}$ | | $7 \mathrm{a} 7 \mathrm{~b}$ | F51 Y128 K165 N166 S167 Y186 R187 I248 G249 A299 | NAP | | 7 pae | M20 L22 L38 V49 I53 C56 K57 R61 Q78 V80 W90 I109 M117 I129 L147 Y149 | O7T | | 8egy | H82 K83 S186 G230 S231 N232 E345 S368 G369 | PLP | | 7qow | S65 R129 D273 H465 | $\mathrm{PO} 4$ | | $7 \mathrm{wmk}$ | E77 L124 R129 S174 T189 Q191 W241 D304 E306 K349 D410 W411 Y486 | PQQ | | $7 \mathrm{pl} 9$ | D607 A608 Y637 M638 Y705 G706 M735 K736 | RET | | $7 \mathrm{yf} 2$ | G153 E174 L175 L209 N210 L211 Y295 | $\mathrm{SAH}$ | | $7 v 6 \mathrm{j}$ | G207 D230 L231 D250 M251 K264 | SAM | | 7 ys6 | D106 C110 N288 | SRO | | $6 \mathrm{w} 8 \mathrm{~m}$ | A22 A23 G70 S110 T111 G112 V113 Y114 | TJY | | $8 g 27$ | S258 D294 K435 R717 | $\mathrm{UDP}$ | | $7 x y k$ | R24 C170 R190 S191 D193 N201 H231 Y233 | UMP | | $8 \mathrm{~g} 3 \mathrm{~s}$ | H224 F228 V249 M250 V253 R263 T266 L267 F270 | YLT | | 8 it 9 | T92 P93 R96 Y108 L109 K216 V228 S229 H231 H232 | ZL6 |
[^1] the atomic coordination dataset.
Simulating 500 million years of evolution with a language model
Figure S17. Alignment improves model generations. pTM, cRMSD distributions of generations from the 98B base model and aligned model for all ligands in the atomic coordination dataset. Each ligand/model pair has 1024 generations.
Figure S18. Randomly selected successful generations from the base model and finetuned model. A random sample of ligands is selected and visualized with the ground truth PDB chain from which the ligand was taken. Solutions produced by ESM3 are diverse, and the finetuned model gives significantly more successes (out of 1024 total samples). putational methods, experimental protocols, results, and post-experiment analyses.
The base ESM3 7B model generates candidate GFP designs for laboratory testing using a single prompt and a chain of thought over sequence and structure tokens. Candidates are filtered and ranked by metrics at several steps in the process. Experiment 1 tests candidates across a range of sequence identity to a template, yielding multiple GFPs including dim hit B8. Experiment 2 consists of designs starting a chain of thought from the sequence of B8, yielding numerous bright GFPs including C10 which we term esmGFP. This section details the computational protocol that generated and selected candidate GFP designs for Experiments 1 and 2, shown in Fig. 4B. Protocols, metrics, and selection conventions are separately introduced and then synthesized in descriptions of the two experiments, at the end of the section.
All candidate GFP designs were created using the base ESM3 7B model with no finetuning. Throughout generation, the model is prevented from decoding cysteine residues.
All candidate GFP designs in Experiment 1 are produced with a chain of thought beginning from a single prompt. The goal of the prompt is to capture essential residue identities and structural features needed for chromophore formation and fluorescence, leaving other degrees of freedom open for the model to generate diverse designs.
Template To this end, we prompt ESM3 with a minimal set of sequence and structure information from 16 residues near the chromophore formation site from a template protein. We select a pre-cyclized intermediate crystal structure from (50), PDB ID 1QY3, as our template. We reverse the chromophore maturation slowing mutation R96A in 1QY3 so the prompt contains Arg96. We subsequently refer to the full sequence and structure of 1QY3 with mutation A96R as 1QY3 A96R or the template.
Sequence prompt The sequence portion of our prompt consists of 7 template residues: Met1, Thr62, Thr65, Tyr66, Gly67, Arg96, and Glu222. Residues 65-67 form the chromophore. Met1 ensures proper start codon placement. Residues 62, 96, and 222 are described in (50) and other works to have key catalytic roles in chromophore formation.
Structure prompt The structure portion of our prompt consists of structure tokens and backbone atomic coordinates taken from 16 template residues at positions 96,222 , and 58-71 (inclusive) which roughly captures the central alpha helix. The unique geometry of the central alpha helix is known to be crucial for chromophore formation (50).
All other positions and tracks in the prompt are masked. The overall prompt length is 229 , matching that of the template. Residue indices are contiguous and begin from 1.
We employ the following procedure to jointly optimize the sequence and structure of designs throughout our experiments: While annealing temperature linearly from 1 to 0 , we perform multiple iterations of first predicting the structure of a designed sequence and subsequently Gibbs sampling each position in the sequence for that predicted structure. In algorithmic form:
Algorithm 15 gibbs_seq_given_struct
Input: ESM3 $f$, sequence $x \in:\{0 . .20\}^{L}$, structure $y$, tem-
perature $t$
for $i=\operatorname{shuffle}(\{1, \ldots, L\})$ do
$x_{i} \sim \exp \left(\log f\left(x_{i} \mid x_{\backslash i}, y\right) / t\right)$
end for
return $\mathrm{x}$
Algorithm 16 joint_optimize
Input: ESM3 $f$, initial sequence $x_{1}$, iterations $I$, initial
temperature $t_{1}$, final temperature $t_{f}$
for $i=1, \ldots, I$ do
$t_{i}=\left(t_{f}-t_{1}\right) \cdot(i /(I-1))+t_{1}$
$y_{i}=$ generate $_{\text {struct }}\left(f, x_{i}\right.$, len $\left.\left(x_{i}\right), T=0\right)$
$x_{i+1}=$ gibbs_seq_given_struct $\left(f, x_{i}, y_{i}, t_{i}\right)$
end for
return $x_{I+1}$
Three variants of gibbsseqgivenstruct in jointoptimize were employed for Experiments 1 and 2. Joint optimization occasionally produces repetitive spans of amino acids when temperature is annealed to low values. Variant 1 and 2 are intended to address this, in differing ways. Variant 3 is an experiment in biasing the logits with a PSSM of known natural GFPs. Half of the candidates in Experiment 2 were produced using Variant 3. This half did not include esmGFP.
$$ \text { logits }^{\prime}=\text { weight } *\left(\text { logits }{\text {cond }}-\text { logits }{\text {uncond }}\right)+\text { logits }_{\text {uncond }} $$
but push away from the logits produced by inputting just 7 residues centered on the position being sampled, with weight 2 and nothing else. All other sequence positions and all other model inputs are left blank.
logits $^{\prime}=2 *\left(\right.$ logits ${\text {cond }}-$ logits $\left.{\text {localseq }}\right)+$ logits ${\text {local_seq }}$
GFP designs are produced and scored by a number of ESM3derived and independent metrics. Unless otherwise noted, designed structures are predicted using ESM3 with only sequence as input, using iterative decoding of structure tokens with temperature 0 and subsequent decoding of backbone coordinates with an older version of the structure token decoder.
The following is an exhaustive list of metrics used. An exact break down of where and how specific metrics are used can be found in Appendix A.5.1.5, Appendix A.5.1.6 and Appendix A.5.1.7.
Template Chromophore Site RMSD is calculated via an optimal alignment (100) of N, C, CA, and inferred $\mathrm{CB}$ atoms at positions $62,65,66,67,96$, and 222 in the predicted structure of a design and the template (crystal) structure.
Template Helix RMSD is calculated in the same way, but for N, C, CA atoms only, at design and template positions 58-71 (inclusive).
1EMA Helix RMSD is a metric proposed in (101). An RMSD is calculated between alpha helix residues in the predicted designed structure and a specific crystal structure of avGFP, PDB ID 1EMA. Our calculation differs slightly from (101). We calculate RMSD for $\mathrm{N}, \mathrm{C}, \mathrm{CA}$ and inferred $\mathrm{O}$ atoms, and consider only positions 60-64 and 68-74 (both ranges inclusive) to exclude chromophore positions 65-67.
Sequence Pseudo-perplexity is calculated as defined in (102). Given a protein sequence, positions are masked one at a time, negative log-likelihoods of input tokens at masked positions are averaged across all positions in the sequence, and the result is exponentiated.
Round-trip Perplexity is calculated for a designed sequence via predicting its structure with ESM3, and then evaluating the perplexity of the sequence given that predicted structure under a single forward pass of ESM3.
$\mathbf{N}$-gram Score is calculated as the $E{\text {ngram }}$ term defined in (10). This score assesses the divergence between the $\mathrm{N}$ gram frequencies of residues in the designed sequence and those found in a background distribution, derived from UniRef50 201803. Specifically, for a function ngram ${i}$ that takes in a sequence $x$ and an $\mathrm{N}$-gram order $i$, and a precomputed distribuion of background $\mathrm{N}$ gram frequencies ngram ${ }{i, b g}$, the score is calculated as:
PSSM A position-specific scoring matrix (PSSM) is constructed from a MSA of 71 natural GFPs (103). Specifically, at positions aligned to our template, frequencies for the 20 canonical amino acids (excluding gaps) are transformed to log odds via dividing by the uniform background $(p(a a)=0.05)$, adding an epsilon of $1 \mathrm{e}-9$, and applying $\log$ base 2 . This produces a matrix of scores of size 229 x 20 .
PSSM score We extract from the PSSM values at (position, amino acid) pairs occurring in an input sequence. These are averaged to produce a score.
N-terminus Coil Count is metric intended to measure structural disorder at the $\mathrm{N}$-terminus of a design. We observed that predicted structures have various levels of disorder in this region. To quantify it for possible filtering, we apply mkdssp (76) to the ESM3-predicted structure of a design, and record how many of the first 12 positions are reported as having SS8 labels in ${\mathrm{S}, \mathrm{T}, \mathrm{C}}$.
Among Experiment 1 and 2, designs are selected for testing by first applying a set of filters, and then selecting the top$\mathrm{N}$ designs according to a score-based ranking. Scores are calculated by summing the values of several metrics, which are each normalized across designs to have zero mean and unit variance and which are negated when appropriate so that lower values are always better.
Common Filters: The following filters are applied in both Experiments 1 and 2.
Common Score Terms: The following score terms are used in both Experiments 1 and 2.
In this experiment, we generate a set of GFP designs for experimental testing with a range of sequence identities to our template. Designs are generated by a chain of thought: From the prompt, ESM3 decodes all masked structure tokens, then all masked sequence tokens. Lastly, sequence and structure tokens are jointly optimized.
Initial Generation: Starting from the prompt, we first generate $38 \mathrm{k}$ structures by decoding masked structure tokens one at a time using a fixed temperature sampled uniformly from the range $(0,1.25)$ for each generation. To focus compute on the most promising structures, we filter according to Template Chromophore Site RMSD $<1 \AA$, yielding $24 \mathrm{k}$ selected structures. We next generate $\approx 4$ sequences for each structure with a temperature uniformly sampled from the range $(0,0.6)$, yielding $92 \mathrm{k}$ total sequences.
Selection: We select a subset of promising initial generations for further optimization by applying Common Filters with $\mathrm{N}$-gram score's threshold modified to $<5.5$, ranking designs according to ${$ Common Score Terms, mean ESM3 pLDDT, mean ESMFold pLDDT, and ESMFold pTM $}$, and selecting the best 40 designs in each interval of 0.1 sequence identity to the template sequence in $[0.2,1.0], 320$ in total.
Joint Sequence Structure Optimization: We then jointly optimize the sequence and structure of designs. Using 30 iterations in each case, we run 5 seeds of optimization with max decoding entropy threshold $=1.5$ and 2 seeds of optimization with negative local sequence guidance $=2.0$, yielding $67 \mathrm{k}$ total designs. Designs from every iteration are included in this pool.
Selection To select a set of designs for laboratory testing, we apply {Common Filters, N-terminus Coil Count $<6}$, rank designs according to ${$ Common Score Terms, ESMFold pTM, 15 * PSSM Score $}$, and select the best 88 designs across 8 buckets of sequence identity to our template among intervals of width 0.1 in range $[0.2,1]$.
In this experiment, we perform further refinement of the dim, distant GFP found in Experiment 1, B10. To produce a diversity of designs, we sweep over a number of settings: two variations of refinement are performed, and 2 selection protocols are used.
Local Joint Optimization: Starting from our dim GFP design, B10, we perform joint_optimize using a full grid sweep of the following sets of settings: Initial temperatures ${0.001,0.01,0.05,0.1,0.5}$, PSSM bias weights ${0,0.01,0.05,0.1,0.5}$, Max decoding entropy thresholds ${0.8,1,1.25,1.5,2.0}$. For each unique settings combination, we use 20 iterations of optimization with 3 seeds, continuing the final step of Gibbs sampling until convergence. After accounting for some distributed system machine failures, this yields $6.3 \mathrm{k}$ total candidate designs.
Selection: We select two sets of 45 designs for laboratory testing via two filters and a shared set of ranking criteria.
For each set, we rank according to ${$ Common Score Terms, 8 * PSSM Score, 15 * 1EMA Helix RMSD} and select 45 designs each for testing.
We designed a custom bacterial expression vector containing an Ampicillin-resistance gene, the BBa_R0040 TetR promoter, the $\mathrm{BBa} B 0015$ terminator, and a Bsa-I golden gate site between the promoter and terminator. GFP designs were codon optimized for E. coli expression and ordered from IDT (Integrated Device Technology Inc.) containing compatible golden gate overhangs. They were then cloned by golden gate assembly into the vector. We evaluated our GFP designs in the E. coli host Mach1.
To evaluate the fluorescence of our GFP designs, we transformed our designs into Mach1 cells. For each of two replicates of a design, a colony was seeded into a $1 \mathrm{~mL}$ TB culture containing $50 \mu \mathrm{g} / \mathrm{mL}$ carbenicillin. Cultures were grown in 96 deep well blocks at $37^{\circ} \mathrm{C}$ in an Infors HT Multitron Shaker with a shaking speed of 1000 RPM for 24 hours. After 24 hours, $1 \mu \mathrm{L}$ of the cultures were diluted in $200 \mu \mathrm{l}$ of $0.2 \mu \mathrm{m}$ filtered DPBS.
Fluorescence intensity of the samples was then quantified at the single cell level using a NovoCyte Quanteon Flow Cytometer (Fig. S19).
The remaining cultures were spun down at $4000 \mathrm{~g}$ for 10 minutes, resuspended and lysed with $300 \mu \mathrm{L}$ lysis buffer (1x bugbuster, $500 \mathrm{mM} \mathrm{NaCl}, 20 \mathrm{mM}$ Tris-HCl pH 8, 10\% glycerol, cOmplete ${ }^{\mathrm{TM}}$, EDTA-free Protease Inhibitor Cocktail), incubated at room temperature on a Belly Dancer Orbital Shaker for 10 minutes, and lysate clarified by centrifugation at $4000 \mathrm{~g}$ for 20 minutes. $100-120 \mu \mathrm{l}$ lysate was transferred to a 96 well black clear-bottom plate, and GFP fluorescence was measured using a Tecan Spark Reader. Fluorescence emission was captured at $515 \mathrm{~nm}$ with a $10 \mathrm{~nm}$ bandwidth and excited with $485 \mathrm{~nm}$ with a $10 \mathrm{~nm}$ bandwidth. Absorbance was captured at $280 \mathrm{~nm}$ with a $3.5 \mathrm{~nm}$ bandwidth to assess total protein content per well. For longer time points, plates containing lysate were sealed and incubated at $37^{\circ} \mathrm{C}$ for up to 7 days prior to measuring fluorescence. GFP fluorescence values were first ratio normalized within a well by their absorbance at $280 \mathrm{~nm}$, and then further ratio normalized across wells using the measured values from a negative control E. coli containing vector without GFP. Data from two replicates was then averaged for (Fig. 4B bottom) and (Fig. 4C).
Overview photos of the plates (Fig. 4B top) were taken with an iPhone 12 mini under blue light illumination from an Invitrogen Safe Imager 2.0 Blue Light Transilluminator.
For excitation spectra, emission was captured at $570 \mathrm{~nm}$ with a $50 \mathrm{~nm}$ bandwidth, while the excitation wavelength was varied from 350 to $520 \mathrm{~nm}$ with a $10 \mathrm{~nm}$ bandwidth. For emission spectra, an excitation wavelength of $430 \mathrm{~nm}$ was used with a $50 \mathrm{~nm}$ bandwidth, while emission was captured at varying wavelengths from 480 to $650 \mathrm{~nm}$ with a $10 \mathrm{~nm}$ bandwidth. Excitation and emission spectra were normalized by their maximum values (Fig. 4C).
Plate overview photographs (Fig. 4B top) were taken over two weeks since the initial lysate was created and over one week after the final plate reader quantification was done, and so possibly show additional brightness from slow chromophore maturing designs. We observed some low level contamination of wells $\mathrm{H} 11$ (vector with no GFP or designs) and H12 (lysis buffer only) in the photograph of Experiment 1 (Fig. 4B top left). Some of this contamination is already visible in well $\mathrm{H} 12$ during the initial plate reader quantification (Fig. 4B bottom left). To address potential contamination concerns we performed an additional replication of B8 and observed a similar level of brightness to Experiment 1 (50x less bright than natural GFPs) (Fig. S20).
Chromophore knockout versions of 1QY3 A96R and esmGFP were created through additional T65G and Y66G mutations. These variants, along with 1QY3 and esmGFP, were synthesized and measured as part of an independent replicate performed by Genscript following the E. Coli based fluorescent plate reader assay described above. Normalization was performed with an OD600 measurement of the cells prior to lysis. Analysis otherwise proceeded as above. Two replicates were performed for each design and results were averaged. Chromophore knockout reduced fluorescence to background levels (Fig. S21).
BLAST nr search: esmGFP's sequence was searched with BLAST's online server using the non-redundant sequences database $\mathrm{nr}$ with all default settings. tagRFP's sequence was taken from the top hit. The exact top hit found was TagRFP [Cloning vector pLX-B2-TagRFP-T, Sequence ID ASG92118.1 and is shown in its entirety in Table S14.
Train set search: MMseqs2 (73), version 15.6f452, was used to search all datasets that ESM3 was trained on at the maximum available expansion level; for cluster resampling datasets all cluster members are searched, not just cluster centers. The goal is to search against every possible sequence that ESM3 may have seen during pre-training. Settings are selected for conducting a high sensitivity search: -s 6 -a --max-seqs 10000 .
To calculate sequence identities involving the two highlighted GFP designs (B8, esmGFP) and select reference proteins, the following procedure is used. MAFFT (104) v7.525 is applied with all default settings to the sequences of B8, esmGFP, the top tagRFP sequence found by BLAST, eqFP578 (from FPBase (105)), the template (PDB ID 1QY3, with mutation A96R), and avGFP (from FPBase). Identities between two sequences are calculated as the number of matching non-gap residues at aligned positions divided by the minimum non-gapped length of the query and target protein. This is the same sequence identity formula used in Appendix A.5.4. Aligned sequences and identities and mutation counts to esmGFP are provided in Table S14.
Figure S19. Flow cytometry data confirms cells expressing esmGFP can be detected at the single cell level. Forward Scatter-Area (FSC-A), a measure of cell size vs Fluorescein Isothiocyanate-Area (FITC-A), a measure of GFP-like fluorescent signal, for expressing 1QY3 A96R, esmGFP, and a negative control that does not express any GFP. A gate was set at the $99.9 \%$ quantile for the negative control data, and the fraction of cells passing the gate were quantified for each sample.
Figure S20. Replication of design B8 and select controls. Results are averages of eight wells across two plates.
Positions in esmGFP are described as internal if they have SASA $<5$ in their predicted structure. SASA is calculated as in Appendix A.2.1.6) from the all-atom structure of esmGFP, predicted with ESM3 7B.
Sequences and metadata of natural and designed fluorescent proteins were obtained from FPBase (105). An initial set of 1000 proteins was filtered to protein which contained the following metadata: a specified parent organism, an amino acid sequence between 200 and 300 residues long, a specified emission maximum, and no cofactors. NCBI taxonomy database was used to obtain taxonomic information about each species. These sequences were further filtered accord-
Figure S21. Chromophore knockout mutations T65G and Y66G reduces fluorescence of both $1 \mathrm{QY} 3$ A96R and esmGFP to background levels.
Figure S22. Sequence identity of esmGFP with natural and designed GFPs from the four major classes found in nature.
ing to keep those that had species found by NCBI and were Eukaryotic but not from Chlorophyta (to exclude Channelrhodopsin like proteins). The 648 sequences that passed these criteria, along with the sequence for esmGFP, were aligned to a multiple sequence alignement using MAFFT and sequence idenity was computed between each pair of sequences as described above. All pairs within and across taxa were considered for (Fig. 4F). All designed sequences were considered to belong to the species annotated as their parent organism.
All 648 used sequences belonged to the Leptocardii (e.g. laGFP), Hexanauplia (e.g. ppluGFP), Hydrozoa (e.g. avGFP), or Anthrozoa (e.g. efasGFP) classes. The sequence identity of esmGFP was computed to each protein in these classes Fig. S22. esmGFP was found to be closest to Anthrozoan GFPs (average sequence identity $51.4 \%$ ) but also shares some sequence identity to Hydrozoan GFPs (average sequence identity $33.4 \%$ ).
To estimate the millions of years of evolutionary distance by time between esmGFP and known fluorescent proteins we built an estimator to go from sequence identity between pairs of GFPs to millions of years (MY) apart. We used the following six Anthozoan species Acropora millepora, Ricordea florida, Montastraea cavernosa, Porites porites, Discosoma sp., Eusmilia fastigiata along with the six GFPs amilGFP, rfloGFP, mcavGFP, pporGFP, dis3GFP, efasGFP respectively. These species and GFPs were chosen because they were annotated in both a recent time calibrated phylogenetic analysis of the Anthozoans (53) and a recent study of GFPs (44). Each of these species contains multiple GFP like sequences including red and cyan FPs. These particular GFPs were chosen as they were annotated to be the main GFP in each species. The millions of years between each species was estimated as twice the millions of years to the last common ancestor annotated in the time calibrated phylogenetic analysis. Using statsmodels (106), a line of best fit was fit between MY and sequence identity. The line was required to pass through a sequence identity of 1.0 and 0 MY. The MY to esmGFP was then estimated using this line and the sequence identity of esmGFP to the nearest known protein.
Please summarize for an expert
AI: This extensive section focuses on the methodology and results of simulating the evolution of Green Fluorescent Protein (GFP) using the ESM3 7B model, aimed at generating novel GFP designs. Here are the key points:
Model and Prompts: The base ESM3 7B model, with no finetuning, was used to generate candidate GFP designs. The model was prompted using specific sequence and structure information from a pre-existing GFP template (PDB ID 1QY3) but with a mutation (R96A reversed to Arg96), focusing on essential residues for chromophore formation and fluorescence.
Joint Sequence and Structure Optimization: The method involved iterative optimization of both the sequence and the backbone structure predictions, aiming to create viable GFP candidates. This optimization used techniques such as Gibbs sampling, with variants including biasing away from repetitive amino acid sequences (Negative Local Sequence Guidance) and a PSSM constructed from natural GFP sequences.
Metrics and Selection Criteria: Various metrics were used to evaluate the generated GFP designs, including template chromophore site and helix RMSD, pseudo-perplexity, and N-gram scores, among others. Candidates were filtered and ranked according to these metrics.
Experiment Phases:
Experimental Methods and Data Analyses: After computational generation, GFP designs were experimentally tested in E. coli. The novelties include custom bacterial expression vectors, golden gate cloning, and fluorescence assays to evaluate the GFP designs.
Results and Observations: The initiative led to the creation of esmGFP, a bright and efficient GFP variant. Notably, esmGFP shows significant differences compared to a template and previously known GFPs yet maintains key characteristics for functionality and fluorescence.
Sequence Searches and Comparisons: Analyses included comparing esmGFP sequences with existing GFPs, revealing esmGFP's distinctiveness from known proteins, thus underscoring the model's ability to generate novel bioactive proteins.
Phylogenetic Analysis: A phylogenetic analysis positioned esmGFP relative to natural fluorescent proteins, estimating its evolutionary distinctness and showing it shares higher sequence identity with Anthozoan GFPs.
This document illuminates the intersection of computational biology and machine learning, showcasing how AI models, particularly the ESM3 model, can be harnessed to accelerate the discovery of novel proteins with desired characteristics, reducing the need for labor and time-intensive laboratory work.
We are releasing the ESM3 source code and model weights of an open model, ESM3-open. ESM3-open is a 1.4Bparameter model we trained without OAS antibody sequences and with precautionary risk mitigations for release to the academic research community.
As part of this release, we follow guidance from the Principles for the Responsible Development of AI for Biological Design (107). We adopted precautionary risk mitigations, described in Appendix A.6.1, and performed risk evaluations, detailed in Appendix A.6.2. Additionally we conducted a review of the risks and benefits of releasing ESM3-open with experts from the scientific community. We provided reviewers access to ESM3-open, along with a detailed technical report on our risk evaluations. We received unanimous feedback from our reviewers that the benefits of releasing the model greatly outweigh any potential risks.
We see this release as a first step and plan to work with the scientific community to continue to improve processes around responsible development. Open models enable the scientific community to better understand and reduce any potential risks of biological design tools. As our understanding develops alongside the capabilities of future models, we plan to continuously improve our evaluation frameworks, safeguards, and mitigation strategies.
As a precaution, we filtered the training data of ESM3-open to minimize model performance on sequences of potential concern while otherwise maintaining performance. We also removed the capability for the model to follow prompts related to viruses and toxins.
Filtering sequences of potential concern. Previous work has shown that the performance of protein language models is closely related to the number of similar sequences present in the training data (5). We therefore removed sequences aligned to potentially-concerning proteins from the training data in order to reduce the capability of ESM3-open on these sequences.
We identified and removed sequences unique to viruses, as well as viral and non-viral sequences from the Select Agents and Toxins List (108) maintained by the CDC and USDA. The U.S. Department of Health \& Human Services recommends filtering based on the Select Agents list as part of their Screening Framework Guidance for Providers and Users of Synthetic Nucleic Acids (109).
| Protein | Sequence
Identity to
esmGFP | Mutations
to esmGFP | Aligned Sequence |
| :---: | :---: | :---: | :---: |
| B8 | 0.93 | 15 | -MSKVEELIKPEMKMKLEMEGEVNGHKFSIEAEGEGKPYEGKQTIKAWSTT-GKLPAW
DILSTSLTYGFRMFTKYPEGLEEHDYFKQSFPEGYSWERTITYEDGATKVTSDISLED
GVLINKIKFKGTNFPSDGPVM-QKKTTGEEPSELITPDPATGGLKGEVKMRLKLEGGG
HLLADFKTTYRSKKKEK-LPLPGVHYVDHTIRNEKAPHPEGKEYVVQYETAVARLA--
------ |
| esmGFP | 1.0 | 0 | -MSKVEELIKPDMKMKLEMEGEVNGHKFSIEAEGEGKPYEGKQTIKAWSTT-GKLPFAW
DILSTSLTYGNRAFTKYPEGLEQHDFFKQSFPEGYSWERTITYDGAAVKVTADISLED
GVLINKVKFKGENFPSDGPVM-QKKTTGEEASTELITPDATGGLKGEVKMRLKLEGGG
HLLADFKTTYRSKKKEK-LPLPGVHYVDHRIVNEKATHPEGKEYMIQYEHAVARLA--
------- |
| tagRFP | 0.58 | 96 | MVSKGEELIKENMHMKLYMEGTVNNHHFKCTSEGEGKPYEGTQTMRIKVVEGGPLPFAF
DILATSFMYGSRTFINHTQGIP--DFEKQSFEEGTWERVVTYEDGGVLTATQDTSLQD
GCLIYNVKIRGVNEPSNGPVM-QKKTLGWEANTEMLY--PADGGLEGRTDMALKLVGGG
HLICNFKTTYRSKKPAKNLKMPGVYYVDHRL--ERIKEADKETYVEQHEVAVARYCDLP
SKLGHKLN |
| eqFP578 | 0.53 | 107 | ----MSELIKENMHMKLYMEGTVNNHHFKCTSEGERKPYEGTQTMKIKVVEGGPLPFAF
DILATSFMYGSKTFINHTQGIP-DDLFKQSFEEGTWERITTYEDGGVLTATQDTSLQN
GCIIYNVKINGVNFPSNGSVM-QKKTLGWEANTEMLY--PADGGLRGHSQMALKLVGGG
YLHCSFKTTYRSKKPAKNLKMPGFHFVDHRL--ERIKEADKETYVEQHEMAVAKYCDLP
SKLGHR-- |
| template | 0.38 | 112 | -MSKGEELFTGVVPILVELDGDVNGHKFSVSGEGEGDATYGKLTLKFICTT-GKLPVPW
PTLVTTLTYGVQCFSRYPDHMKQHDFKSAMPEGYVQERIISKDDGNYKTRAEVKFEG
DTLVNRIELKGIDFKEDGNILGHKLEYNYNSHNVYITADKQKNGIKANFKIRHNIEDGS
VQLADHYQQNTPIGDGP-VLLPDNHYLSTQSALSKDPN-EKRDHMVLLEFVTAAGI-- |
| avGFP | 0.36 | 146 | -MSKGEELFTGVVPILVELDGDVNGHKFSVSGEGEGDATYGKLTLKFICTT-GKLPVPW
PTLVTTFSYGVQCESRYPDHMKQHDFFKSAMPEGYVEERTIFKRDGNYKKRAEVKFEG
DTLVNRIELKGIDFKEDGNILGHKLEYNYNSHNYYMADKQKNGIKVNFKIRHNIEDGS
VQLADHYQQNTPIGDGP-VLLPDNHYLSTQSALSKDPN-EKRDHMVLLEFVTAAGITHG
MDELYK-- |
Table S14. Multiple sequence alignment of select GFP designs (B8, esmGFP) and reference proteins. Template is the full sequence of our template structure (PDB ID 1QY3), with chromophore slowing mutation A96R removed. tagRFP is the full sequence of the top hit returned by BLAST search of the nonredundant database $\mathrm{n} r$, avGFP and eqFP578 are from FPBase. Sequence identities for GFP designs are in general calculated as the number of non-gap matches at aligned positions, divided by the minimum length of the query and target ungapped sequences. Here, only sequence identities to esmGFP are shown. Similarly, the number of mutations to esmGFP are calculated as the number of mismatches at aligned positions where esmGFP does not have a gap.
Figure S23. ESM3-open is a powerful predictor of structure and function trained for open release. A: Structure Prediction ESM3open (blue) is competitive with ESMFold (orange) on structure prediction as measured by LDDT on CAMEO and CASP14/15. See Appendix A.3.4 for details on this evaluation. B: Representation Learning ESM3-open (blue) is competitive with ESM2-3B (orange) on representation learning as measured by contact prediction $\mathrm{P} @ \mathrm{~L}$ for finetuned representations. See Appendix A.3.3 for details on this evaluation. C: Function Keyword Prediction. ESM3-open function prediction performance, as measured by Mean Average Precision across function keywords. ESM3-open achieves 0.81 precision across all keywords, and 0.89 for the top $1 \mathrm{~K}$ most prevalent keywords in the validation set (CAMEO). We use the same evaluation framework as in Appendix A.1.8.2.2. We report both the macro and micro averages as in Fig. S8. In each of the preceding evaluations, the data mitigation minimally impacted performance, as compared to a compute-matched model without data mitigations (hatched blue). D: Zero-shot Fitness Prediction. Fitness prediction performance as measured by correlation (Spearman $\rho$ ) across 217 Deep Mutational Scanning datasets collated in ProteinGym. Left and right subplots indicate viral (left) and non-viral (right) DMS datasets. The four columns per group indicate different models. ESM3-open performs substantially worse than EVMutation (purple) on viral fitness prediction, while being competitive with ESM2 (orange) on non-viral fitness prediction. Viral fitness prediction was substantially impacted by the data mitigation, while non-viral fitness prediction was not (hatched blue).
To filter data, we create two denylists: the Viral Denylist and the Select Agent Denylist. We then remove all sequences from the training set that are detected to align to those in the denylists by MMseqs2 at or above a given sequence identity threshold.
To create the Viral Denylist, we identify $\sim 4 \mathrm{M}$ sequences that are annotated as viral in UniProt and align almost exclusively to other viral sequences in UniProt. This gives us a procedure that removes viral proteins with both high sensitivity and specificity (as measured by UniProt taxonomic annotations). To create the Select Agents Denylist we identify all sequences in UniProt belonging to organisms on the Select Agents and Toxins List (108). This process gives us $147 \mathrm{~K}$ non-viral sequences and $40 \mathrm{~K}$ additional viral sequences.
For each denylist, MMseqs was used to query against the full set of training databases, (including PDB, UniRef, MGnify, and JGI) and all hits were removed from the training set. This filter removes a total of $10.6 \mathrm{M}$ sequences across all training sets.
Removal of keywords of concern. There are a number of keyword prompts associated with viruses and toxins that we aim to remove. We first identify a list of harmful keywords with the following steps:
Of the original 68,103 keywords that ESM3 is trained with, this filter removes a total of 9,462 (14\%), creating a new vocabulary of 58,641 keywords.
The function vocabulary is defined via vectors representing Term Frequency Inverse Document Frequency (TF-IDF) which are then tokenized using Locality Sensitive Hashing (LSH), as previously described in Appendix A.1.8. To remove flagged keywords, they are first removed from the TF-IDF vocabulary by removing the entries corresponding to flagged keywords. This reduces the TF-IDF vector size to 58,641 . The LSH tokenization is defined by 64 hyperplanes, each defined in the TF-IDF space, i.e. a Euclidean space with one dimension per keyword. We redefine the hyperplanes to be in the reduced space by removing the dimensions corresponding to the flagged keywords. This per- manently removes the information required for tokenization of the flagged keywords. This mitigation is highly selective and does not change the tokenization for any non-flagged keywords.
In the section below, we outline our evaluations of ESM3open performance. When appropriate, we compare ESM3open to either existing open models, (e.g. ESM2 or ESMFold), or to a compute-matched version of ESM3-open, trained without any data mitigations.
Structure Prediction In Fig. S23A, we show that ESM3open achieves competitive performance on structure prediction as measured by LDDT on CASP14, 15 and CAMEO, showing very slight degradation from our compute-matched 1.4B model without data filtering. The evaluation framework is described in Appendix A.3.4.
We also measure the ability of ESM3 to predict the structure of a subset of viral proteins. In Fig. S23A we evaluate structure prediction on a set of structures derived from viruses that were purged from the PDB training set. For the chains in PDB that were $>70 \%$ sequence identity hits to the Viral Denylist, we cluster at $40 \%$ sequence identity and then select the longest chain (with length $\leq$ 1024) from each cluster. ESMfold and ESM3-open achieved an average LDDT of 0.66 and 0.63 , respectively, on the viral structures. Without the data mitigation, a compute-matched ESM3-open would have achieved an average LDDT of 0.66. This is substantially worse than the performance on generic structure prediction on CAMEO, and CASP14, where ESMFold achieved an average LDDT of 0.86 and 0.73, and ESM3open achieved an average of LDDT of 0.83 and 0.70.
Representation Learning. ESM3-open achieves strong performance on representation learning, slightly outperforming ESM2 (3B) on contact prediction as measured by precision at $\mathrm{L}(\mathrm{P} @ \mathrm{~L})$ on structures derived from CASP14/15, and CAMEO, see Fig. S23B. The evaluation framework is described in Appendix A.3.3.
Function Keyword Prediction. ESM3-open is able to predict function keywords for proteins in a validation set derived from UniRef and annotated with InterProScan, see Fig. S23C. ESM3-open achieves a Mean Average Precision for all keywords of 0.81 (macro average), and a precision of 0.89 (micro average) for the top 1000 keywords, discarding common terms such as "the". The evaluation framework is the same as that described in Appendix A.1.8.2.2.
Zero-shot Viral Fitness Prediction. We measure the ability of ESM3 to identify viable sequences and understand the effects of mutations on viral proteins. The evaluation consists of the single mutant variants from 217 Deep Mutational Scanning (DMS) datasets collected in ProteinGym (110). This includes 28 DMS landscapes from viral proteins and 189 from other proteins. We evaluate the correlation (Spearman $\rho$ ) between the predicted variant effect and measured variant effect. The predicted variant effect is measured as the difference between the logit value for the variant allele and the logit value of the wildtype allele at a given masked position (16).
First, we compare the performance of ESM3-open to a compute-matched version of ESM3-open which did not undergo any data filtering. Applying data filtering as a mitigation reduces average Spearman $\rho$ performance on viral fitness prediction from 0.28 (ESM3-small) to 0.17 (ESM3-open), while performance on non-viral proteins is not adversely affected, changing from 0.46 (ESM3-small) to 0.45 (ESM3-open). We also compare the performance of ESM3-open to existing open model baselines. Fig. S23D assesses performance relative to the EVMutation (111) baseline. EVMutation is a Markov Random Field model (not deep learning-based) trained on a multiple sequence alignment of the target protein. BLOSUM62 is a baseline based on amino acid substitution frequencies. After mitigations, ESM3-open performance on viral landscapes is low compared to EVMutation and on-par with BLOSUM62.
Please summarize for an expert
AI: The release of the ESM3-open, a 1.4 billion parameter model for protein structure and function prediction, marks a significant step in the direction of open scientific resources. The model is crafted without using Open Antibody Sequences (OAS) and incorporates several risk mitigations to align with the responsible development of AI in biological design. The development team has undertaken a comprehensive risk assessment and collaborated with experts to ensure that the benefits of releasing the model surpass potential risks.
The key risk mitigation strategies include:
Data Filtering: The training data for ESM3-open was filtered to minimize the model's performance on potentially dangerous sequences (e.g., viruses, toxins) by removing sequences that closely match known sequences of concern or are identified on the CDC and USDA's Select Agents and Toxins List.
Keyword Filtering: The team removed prompts related to viruses and toxins from the model's capabilities by identifying and excluding harmful keywords from the model's training data. This reduces the risk of the model being used to generate sequences associated with pathogenic organisms or toxins.
Despite these mitigations, ESM3-open maintains competitive performance in several areas:
Structure Prediction: ESM3-open shows nearly matching performance with a state-of-the-art model, ESMFold, and its non-mitigated counterpart in predicting protein structures, evaluated through LDDT scores on datasets like CASP14/15 and CAMEO.
Representation Learning: It slightly outperforms ESM2-3B in representation learning, notably in contact prediction tasks, indicating the model's robustness in capturing and extrapolating in-depth features from protein sequences.
Function Keyword Prediction: The model proficiently predicts function keywords for proteins, achieving high precision across a validation set derived from UniRef and annotated with InterProScan.
Zero-Shot Fitness Prediction: Although data filtering reduced ESM3-open's performance on viral fitness prediction (compared to non-mitigated versions and a dedicated computational model, EVMutation), it remains competitive in non-viral protein predictions, underscoring its utility in understanding genetic variations' impact.
By removing sequences and capabilities related to harmful agents and relying on expert evaluations, the developers of ESM3-open intend for it to serve as a foundation for more responsible advancements in AI-driven biological research. The project exemplifies a balanced approach to innovation in sensitive domains, stressing the importance of community engagement and continuous improvement in the technologies that intersect AI and life sciences.
S1 The ESM3 architecture . . . . . . . . . . . 22
S2 Geometric Attention . . . . . . . . . . . . 25
S3 Structure tokenizer reconstruction quality . 32
S4 Visualization of structure tokenizer reconstructions
33
S5 Visualization of local neighborhoods which map to the same learned structure token . . 34
S6 pTM and pLDDT calibration . . . . . . . . 35
S7 Schematic of function tokenization . . . . . 35
S8 Function prediction benchmarking results . 36
S9 Visualization of noise schedules used . . . . 41
S10 Scaling curves for structure prediction . . . 43
S11 Conditional and unconditional Scaling behavior for each track 45
S12 Distribution of pTM and pLDDT . . . . . . 45
S13 Unconditional generation of high-quality and diverse proteins using ESM3 . . . . . . 47
S14 Generation of sequences using chain-of- thought . . . . . . . . . . . . . . . . . . . 48
S15 Prompting ESM3 to generalize beyond its training distribution 50
S16 Multimodal protein editing with ESM3 . . . 54
S17 Alignment improves model generations . . 57
S18 Randomly selected successful generations from the base model and finetuned model . 58
S19 Flow cytometry data confirms cells expressing esmGFP can be detected at the single cell level
63
S20 B8 Replication ….. 63
S21 Chromophore knockout mutations ….. 63 S22 Sequence identity of esmGFP ….. 64 S23 ESM3-open is a powerful predictor of struc- ture and function trained for open release ….. 66
S1 Parameter details for different model configurations
24
S2 Training details for stage 2 training of an all-atom structure token decoder ….. 31 S3 Pre-training dataset statistics ….. 40 S4 Pre-training unique token statistics ….. 40 S5 Data augmentation and conditioning infor- mation applied to each dataset ….. 40 S6 Noise Schedules and Dropout Probabilities ….. 41 S7 Precision @ L ….. 44 S8 Protein structure prediction results ….. 44 S9 Negative log-likelihood of each track condi- tioned on other tracks ….. 44 S10 Functional motif definitions for conserved region ….. 50
S11 InterPro tags extracted from CAMEO test set proteins for prompting with fold specification
S12 Novelty and designability metrics. . . . . . 52
S13 Atomic coordination dataset . . . . . . . . 56
S14 Multiple sequence alignment of select GFP designs (B8, esmGFP) and reference proteins 65
[^0]: ${ }^{*}$ Equal contribution ${ }^{1}$ EvolutionaryScale, PBC ${ }^{2}$ Arc Institute ${ }^{3}$ University of California, Berkeley ${ }^{4}$ Work done during internship at EvolutionaryScale, PBC ${ }^{\dagger}$ Correspondence to $<$ arives@evolutionaryscale.ai>.
Preview 2024-06-25. Pending submission to bioRxiv. Copyright 2024 by the authors.
[^1]: Table S13. Atomic coordination dataset. Selected PDBs and coordinating residues (along with binding ligand) for each protein sample in