Über uns
Jennifer Marsman is the Principal Engineer of Microsoft’s “AI for Earth” group, where she…
Articles by Jennifer
-
Inside the New Microsoft, Where Lie Detection is a Killer App
Inside the New Microsoft, Where Lie Detection is a Killer App
By Jennifer Marsman
Activity
-
One of the huge questions of our time: what will be the economic impact of generative AI, and how will it affect jobs? Last night, I was listening…
One of the huge questions of our time: what will be the economic impact of generative AI, and how will it affect jobs? Last night, I was listening…
Shared by Jennifer Marsman
Licenses & Certifications
Publications
-
RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture
There are two common ways in which developers are incorporating proprietary and domain-specific data when building applications of Large Language Models (LLMs): Retrieval-Augmented Generation (RAG) and Fine-Tuning. RAG augments the prompt with the external data, while fine-Tuning incorporates the additional knowledge into the model itself. However, the pros and cons of both approaches are not well understood. In this paper, we propose a pipeline for fine-tuning and RAG, and present the…
There are two common ways in which developers are incorporating proprietary and domain-specific data when building applications of Large Language Models (LLMs): Retrieval-Augmented Generation (RAG) and Fine-Tuning. RAG augments the prompt with the external data, while fine-Tuning incorporates the additional knowledge into the model itself. However, the pros and cons of both approaches are not well understood. In this paper, we propose a pipeline for fine-tuning and RAG, and present the tradeoffs of both for multiple popular LLMs, including Llama2-13B, GPT-3.5, and GPT-4. Our pipeline consists of multiple stages, including extracting information from PDFs, generating questions and answers, using them for fine-tuning, and leveraging GPT-4 for evaluating the results. We propose metrics to assess the performance of different stages of the RAG and fine-Tuning pipeline. We conduct an in-depth study on an agricultural dataset. Agriculture as an industry has not seen much penetration of AI, and we study a potentially disruptive application - what if we could provide location-specific insights to a farmer? Our results show the effectiveness of our dataset generation pipeline in capturing geographic-specific knowledge, and the quantitative and qualitative benefits of RAG and fine-tuning. We see an accuracy increase of over 6 p.p. when fine-tuning the model and this is cumulative with RAG, which increases accuracy by 5 p.p. further. In one particular experiment, we also demonstrate that the fine-tuned model leverages information from across geographies to answer specific questions, increasing answer similarity from 47% to 72%. Overall, the results point to how systems built using LLMs can be adapted to respond and incorporate knowledge across a dimension that is critical for a specific industry, paving the way for further applications of LLMs in other industrial domains.
-
Distributed Inference and API Hosting for an Image Analysis Service: A Case Study on Land Cover Mapping
American Geophysical Union
The AI for Earth Land Cover Mapping project aims to produce a high-resolution land cover map of the United States using machine learning and computer vision. In addition to the core algorithmic challenges facing related to training ML models on large geospatial image sets, additional challenges arise when trying to expose the resulting models to end users (e.g., geospatial analysts). Developing APIs to make ML models widely available requires expertise distinct from the underlying geospatial…
The AI for Earth Land Cover Mapping project aims to produce a high-resolution land cover map of the United States using machine learning and computer vision. In addition to the core algorithmic challenges facing related to training ML models on large geospatial image sets, additional challenges arise when trying to expose the resulting models to end users (e.g., geospatial analysts). Developing APIs to make ML models widely available requires expertise distinct from the underlying geospatial processing and machine learning, and even for experienced developers, maintaining a real-time inference system is both cumbersome and expensive. Consequently, models described in the literature or made available in binary form often remain inaccessible to end users, who may be unfamiliar with machine learning and/or lack the computational resources to run models at scale.
In this presentation, we will use our Land Cover Mapping work as a case study to present one path to making a trained machine learning model available as a spatial querying API. We will discuss the use of Azure Machine Learning Service to run parallel inference and generate a nationwide land cover map that can be served without the complexities of a real-time inference system, and we will discuss the use of the Microsoft AI for Earth open-source API framework to turn that set of cached results into a Web-based API.
We hope this case study can provide a template for other scientists whose primary focus is on geospatial analysis techniques, but who also seek opportunities to expose those techniques to a broad audience. -
Self-Aware Synthetic Forces: Improved Robustness Through Qualitative Reasoning
Proceedings of 2002 Interservice/Industry Training Simulation and Education Conference
Beard, J., Nielsen, P., Kiessel, J. (2002) "Self-Aware Synthetic Forces: Improved Robustness Through Qualitative Reasoning", Proceedings of 2002 Interservice/Industry Training Simulation and Education Conference, December, 2002. Orlando, FL.
-
Failure Recovery: A Software Engineering Methodology for Robust Agents
Proceedings of the 2002 SELMAS Conference
Kiessel, J., Nielsen, P., Beard, J. (2002) "Failure Recovery: A Software Engineering Methodology for Robust Agents", Proceedings of the 2002 SELMAS Conference, May, 2002. Orlando, FL.
-
Robust Behavior Modeling
Proceedings of the 11th CGF Conference
Nielsen, P., Beard, J., Kiessel, J., and Beisaw, J. (2002) "Robust Behavior Modeling", Proceedings of the 11th CGF Conference, May, 2002. Orlando, FL.
Patents
-
Forming Intent-Based Clusters and Employing Same by Search
Issued US 7657519
A method is provided for analyzing a plurality of search sessions to identify intent-based clusters therein. Each session comprises at least one received query from a user and a corresponding set of returned search results, and each set of search results includes, or refers to at least one piece of content. Each cluster represents a group of similar search sessions that are perceived as representing a common purpose and that can be mapped to a common set of search results. In the method, for…
A method is provided for analyzing a plurality of search sessions to identify intent-based clusters therein. Each session comprises at least one received query from a user and a corresponding set of returned search results, and each set of search results includes, or refers to at least one piece of content. Each cluster represents a group of similar search sessions that are perceived as representing a common purpose and that can be mapped to a common set of search results. In the method, for each search session, each received query thereof, the corresponding set of search results, and whether any particular piece of content of the search results was acceptable to the user as responsive to the corresponding search session are identified.
Other inventorsSee patent -
Analyzing Operational and Other Data From Search System Or The Like
US MSFT-4162/308130.1
Honors & Awards
-
Top Tech Influencers driving the Environmental Sustainability Debate
Onalytica
I was ranked #15 on a list of “Top Tech Influencers driving the Environmental Sustainability Debate”. https://onalytica.com/blog/posts/tech-and-environmental-sustainability-top-40-influencers/
-
#2 Most Influential Developer in AI
Onalytica
http://www.onalytica.com/blog/posts/top-influential-developers-in-ai-cloud-iot-cybersecurity-and-vr-ar-mr/
-
Top 10 Influential Female Developers on StackOverflow
Traackr
http://www.traackr.com/blog/10-influential-female-developers-to-know-and-love-on-stack-overflow
-
"Best in Role" for Technical Evangelism
Microsoft
In 2016, I received the "Best in Role" distinction for my work as a developer evangelist at Microsoft.
-
Top 100 Most Influential Individuals in Artificial Intelligence & Machine Learning
Onalytica
On March 3, 2016, I was named one of the "top 100 most influential individuals" in artificial intelligence and machine learning by Onalytica. http://www.onalytica.com/blog/posts/artificial-intelligence-machine-learning-top-100-influencers-and-brands/
Languages
-
Englisch
-
Recommendations received
8 people have recommended Jennifer
Join now to viewOther similar profiles
Gemeinsame Artikel erkunden
We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.
Explore MoreOthers named Jennifer Marsman
-
Jennifer I. Marsman
Director of Operations
-
Jennifer Marsman
mental supervisor at Harvest Construction, LLC
2 others named Jennifer Marsman are on LinkedIn
See others named Jennifer Marsman