Manage Azure OpenAI Service using the Azure CLI

I was working on a project recently where we were using the Azure OpenAI service quite heavily. As part of creating the DevOps pipelines for the project, we had to look into automating the management of the Azure OpenAI service. Turns out this functionality is possible with the Azure CLI however it is available under the Cognitive Services module which can be a bit tricky to find. So here is a quick blog post detailing some of the more frequently used operations for the Azure OpenAI service through the Azure CLI:

For a full set of operations, please see the Microsoft docs: https://learn.microsoft.com/en-us/cli/azure/cognitiveservices?view=azure-cli-latest

Teams tab fails to load in the new Microsoft Teams Desktop client

The new Microsoft Teams Desktop client was made generally available for Windows and Mac recently. The good news is that the new client provides feature parity for 3rd party apps like Focusworks AI giving customers a choice of using their preferred Teams client to access the apps.

However, if you have a custom built Microsoft Teams tab or a task module as part of your solution, and find that it fails to load in the new Microsoft Teams client, there might be a specific reason for it. 

And since there is no way to invoke the Developer tools in the new Teams desktop client yet (November 2023), the experience can get a bit frustrating. 

In my case, I have a custom React/TypeScript based tab which is using the @microsoft/teams-js library to interact with Teams. 

Since teams tabs are just HTML pages, we need to make sure that the page is being loaded inside Teams before continuing to execute the code. To do that we can use the context.app.host.name property and check that the value was "teams" before moving ahead.

However, with the new desktop client my tab was failing to load. After a bit of digging around I realised that the new Teams desktop client has an entirely different host name property and the value is "teamsModern" as mentioned here: https://learn.microsoft.com/en-us/javascript/api/%40microsoft/teams-js/hostname?view=msteams-client-js-latest

So changing my code to include the new value as well worked!

Hope this saves you some debugging time!

Connect an OpenAI chat bot to the internet using Bing Search API

In the previous post, we saw what is OpenAI function calling and how to use it to chat with your organization's user directory using Microsoft Graph. Please have a look at the article here: Chat with your user directory using OpenAI functions and Microsoft Graph

In this post, we will implement function calling for a very common scenario of augmenting the large language model's responses with data fetched from internet search.

Since the Large Language model (LLM) was trained with data only up to a certain date, we cannot talk to it about events which happened after that date. To solve this, we will use OpenAI function calling to call out the Bing Search API and then augment the LLM's responses with the data returned via internet search.

This pattern is called Retrieval Augmented Generation or RAG. 

Let's look at the code now on how to achieve this. In this code sample I have used the following nuget packages:

https://www.nuget.org/packages/Azure.AI.OpenAI/1.0.0-beta.6/

https://www.nuget.org/packages/Azure.Identity/1.10.2/

The very first thing we will look at is our function definition for informing the model that it can call out to external search API to search information:

In this function we are informing the LLM that if it needs to search the internet as part of providing the responses, it can call this function. The function name will be returned in the response and the relevant parameters will be provided as well.

Next, let's see how our orchestrator looks. I have added comments to each line where relevant:

This code is responsible for handling the chat with OpenAI, calling the Bing API and also responding back to the user based on the response from internet search. 

Next, let's have a look at the code which calls the Bing API based on the parameters provided by the LLM. Before executing this code, you will need to have created Bing Web Search API resource in Azure. Here is more information on it: https://learn.microsoft.com/en-us/bing/search-apis/bing-web-search/overview

The Bing Web Search API key can be found in the "Keys and Endpoint" section on the Azure resource:

Here is the code for calling the Bing Search API:

In this code, we are calling the Bing Web Search REST API to get results based on the search query created by the LLM. Once the top 3 results are fetched we are getting the text snippets of those results, combining them and sending it back the LLM. 

We are only getting the search result snippets to keep this demo simple. In production, ideally you will need to get the Url of each search result and then get the content of the page using the Url.

Finally, lets have a look at our CallChatGPT function which is responsible for talking to the Open AI chat API:

This code defines the OpenAI function which will be included in our Chat API calls. Also, the user's question is sent to the Chat API to determine if the function needs to be called. This function is also called again after the response from the Bing Web Search API is fetched. At that time, this function contains the search results and uses them to generate an output in natural language.

This way, we can use Open AI function calling together with Bing Web Search API to connect our chat bot to the internet!

Chat with your user directory using OpenAI functions and Microsoft Graph

Ever since OpenAI function calling was released, I have been incredibly fascinated by it. To me, it is as big a game changer as ChatGPT itself. 

With function calling, we are no longer limited by the data which was used to train the Large Language Model (LLM). We can call out to external APIs, protected company data and other business specific APIs and use the data to supplement the responses from the LLM. 

To know more about function calling specifically with the Azure OpenAI service, check out the Microsoft docs: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/function-calling

In this post, let's have a look at how we can leverage OpenAI function calling to chat with our user directory and search for users in natural language. To make this possible we will use the Microsoft Graph to do the heavy lifting. 

This is what we want to achieve: The user asks a question about the people directory in natural language, the LLM is able to transform the question to code which the Microsoft Graph understands and the LLM is again able to transform the response from the Microsoft Graph back to natural language.

On a high level, our approach can be summarised as follows:

1. Define the OpenAI functions and make them available to the LLM 

2. During the course of the chat, if the LLM thinks that to respond to the user, it needs to call our function, it will respond with the function name along with the parameters to be sent to function.

3. Call the Microsoft Graph user search API based on the parameters provided by the LLM.

4. Send the results returned from the Microsoft Graph back to the LLM to generate a response in natural language.

Alright, let's now look at the code. In this code sample I have used the following nuget packages:

https://www.nuget.org/packages/Azure.AI.OpenAI/1.0.0-beta.6/

https://www.nuget.org/packages/Microsoft.Graph/5.30.0/

https://www.nuget.org/packages/Azure.Identity/1.10.2/

The very first thing we will look at is our function definition:

In this function we are informing the LLM that if needs to search any users as part of providing the responses, it can call this function. The function name will be returned in the response and the relevant parameters will be provided as well. 

The enums in the officeLocation and department parameter will instruct the LLM to only return those values even if user asks a slightly different variation in their question. We can see an example of this in the gif above. Even if the question asked contains words like "devs" and "NY", the LLM is able to determine and use the terms "Engineering" and "New York" instead.

Next, let's see how our orchestrator looks. I have added comments to each line where relevant:

There is a lot to unpack here as this function is the one which does the heavy lifting. This code is responsible for handling the chat with OpenAI, calling the MS Graph and also responding back to the user based on the response from the Graph.

Next, let's have a look at the code which calls the Microsoft Graph based on the parameters provided by the LLM. 

Before executing this code, you will need to have created an App registration with a clientId and clientSecret. Here is how to do that: https://learn.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app

Since we are calling the Microsoft Graph /users endpoint with application permissions, the app registration will need a minimum of the User.Read.All application permission granted.
https://learn.microsoft.com/en-us/graph/api/user-list?view=graph-rest-1.0&tabs=http 

This code get the parameters sent from the LLM and uses the Microsoft Graph .NET SDK to call the /users/search endpoint and fetch the users based on the officeLocation, department or jobTitle properties.

Once the users are returned, their displayName value is concatenated into a string and returned to the orchestrator function so that it can be sent again to the LLM.

Finally, lets have a look at our CallChatGPT function which is responsible for talking to the Open AI chat api. 

This function defines the Open AI function which will be included in our Chat API calls. Also, the user's question is sent to the API to determine if the function needs to be called. This function is also called again after the response from the Microsoft Graph is fetched. At that time, this function contains the details fetched from the Graph to generate an output in natural language. 

This way, we can use Open AI function calling together with Microsoft Graph API to chat with your user directory. 

Building a Microsoft Teams app: Posting a message to Teams on behalf of the current user

If you are building a Teams app and want to integrate with Teams channel messages, this post will be helpful to you. Specifically, we will be looking at posting a Teams channel message, from the app, on behalf of the currently logged in user.  For example, there could be a scenario where an event occurs within the Teams app and as a result a message needs to be posted to a Teams channel but instead of it coming from a bot, it needs to be posted by the logged in user's account.

Let's see how this can be done. 

1. Azure AD app and Permissions

First thing we need is an Azure AD app setup with the Microsoft Graph ChannelMessage.Send delegated permission. This permission is needed for posting messages to Teams using the current user credentials.

I should mention setting up the right permissions is part of a larger configuration in the Azure AD app needed for Single Sign On (SSO) setup in Teams app. You can see the full configuration here: Register your tab app with Azure AD - Teams | Microsoft Learn 

2. Current user's id token from Teams JS SDK v2 

Once the permissions are setup, we need to setup our frontend so that it can grab the current user's id token from Microsoft Teams. More info on Azure AD id tokens here: Microsoft identity platform ID tokens - Microsoft Entra | Microsoft Learn 

Although this token is available from Teams JS SDK v2, it cannot itself be used to make graph calls. We need to exchange it for a Microsoft Graph access token. For this we will send the id token to our backend:

3. Getting Microsoft Graph delegated access token and posting message to Teams

It is recommended to do the token exchange as well as any further Graph calls from the backend of your app instead passing the Graph access token back to the frontend and making the calls from there:

In the code we first graph the id token from the Authorization header, then exchange the id token for a Microsoft Graph access token, then finally we are able to make a Graph call to post a message to Teams as the current user.

Hope this helps!

Building a custom Microsoft Teams tab: Show a native loading indicator

When building a custom Microsoft Teams tab with Teams manifest v1.7+, there is an option to natively show a loading indicator in Teams before our app loads. This can be helpful if your app needs to fetch any data before the first load.

There are two parts to showing the loading indicator: 

1. The showLoadingIndicator property in the Teams manifest needs to be set to true.

2. When the tab loads, the app.initialize() and app.notifySuccess() methods of the Teams JS SDK v2 should be called to indicate to Teams that the app is ready to load and to hide the loading indicator.

Now there are some "interesting" things to know about this. If you are using the Teams Developer Portal to configure your Teams app, then as of now the showLoadingIndicator property is set to true by default and there is no way to change this in the portal. You will have to download the app manifest and make the necessary changes in the json manually.

If you are just starting with Microsoft Teams development and are unaware of the loading indicator option, the error message shown on the tab is not much help in figuring out the issue. It simply says:

"There was a problem reaching the app"

This is because the showLoadingIndicator property has been set to true by the Teams Developer portal and we don't yet have our tab calling the app.initialize() and app.notifySuccess() methods of the SDK. So, Teams thinks that our app has failed to load and shows the error message.

There are two possible solutions to fixing this problem:

1. Set the showLoadingIndicator property to false manually in the manifest json. But the drawback to this approach then, is that no native loading indicator would be shown, and the users might have to stare at a blank screen until the content loads.

2. Let the showLoadingIndicator property be set to true and then make sure we call the app.initialize() and app.notifySuccess() methods of the SDK.

Manifest.json:

Tab react code:

There are further options possible with the loading indicator like hiding the loading indicator even if the app continues to load in the background and explicitly specifying loading failure. To see more details about those, the Microsoft docs are a good place to start: Create a content page - Teams | Microsoft Learn

Hope this helps!

Partially update documents in Azure Cosmos DB

I have been working with Cosmos DB for a while now and until recently, there was one thing which always annoyed me: When updating a JSON document stored in a container, there was no way to only modify a few selected properties of the document. 

The entire JSON document had to be fetched by the client first, then locally replace the properties to be updated, and then send the entire document back to Cosmos DB and replace the previous version of the document. 

This was always a challenge because first, it added more work for developers and second, there could be concurrency issues if multiple clients could be downloading multiple copies of the document and updating the data and sending back their copy.

But fortunately, now it's possible to partially update a document in Cosmos DB and basically do a PATCH operation while only sending the properties to be changed over the wire. 

So in this post, let's have a look at the different ways in which we can partially update documents in Cosmos DB:

Setting up by creating a Cosmos DB document

First, we will create our sample document using code. I should mention I am using v3.30.1 of Azure Cosmos DB .NET core package from nuget: Microsoft.Azure.Cosmos

As you can see this is a simple document representing a User object with a few attached properties:

Now in order to only change a few properties in the object, we need to use the Partial document update feature in Azure Cosmos DB 

Update document properties

Now lets have a look at how we can modify and add properties to the User object:

In the code, we are updating an existing property of the document as well as adding a new property. Also, we are only sending the properties to be modified over the wire. 

We can also send both properties in a single operation by adding the operations to the patchOperations array.

Update elements in array

Just like document properties, we can also update single elements in an array in the document. Elements can be updated, added, added at a specific index and also removed from an array:

Update objects and their properties

The properties of a JSON document can themselves be objects as well. The Azure Cosmos DB patch operations can also be used to update properties of specific objects as well as modifying the objects themselves.

Hope this helps! For more details, do have a look at the Microsoft docs for Azure Cosmos DB partial updates: https://docs.microsoft.com/en-us/azure/cosmos-db/partial-document-update