CoPilot feature gift pack! Microsoft brings personal AI assistant into teams, GPT-4o now powers Azure, browser video real-time translation

From the author of this article: Li Dan

Source: Hard AI

After announcing the integration of Copilot into the Windows system with the Windows 11 AI PC one day ago, Microsoft has now introduced a variety of features for Copilot, further promoting the integration of the advanced artificial intelligence (AI) assistant with Microsoft's business and its partners.

At the annual Build developer conference, which opened on Tuesday, May 21st, Eastern Time, Microsoft CEO Satya Nadella mentioned two dreams that Microsoft has had for decades: one is to make computers understand humans, rather than humans being forced to understand computers, and the other is to help humans reason, plan, and effectively utilize all information in a world of increasing data. He believes that the current wave of AI can provide answers for Microsoft to achieve these two dreams.

Nadella then focused on Copilot. He mentioned the Windows Copilot Library, which includes a locally available API for Copilot, with over 40 pre-built edge AI models, showcasing how the new PC Copilot+ PC fully integrates with personal computers (PCs) and AI applications.

On Monday, Microsoft introduced over 40 AI models in the Windows Copilot Runtime. On Tuesday, Nadella likened the Windows Copilot Runtime to the heavyweight Windows API Win32, as it makes it easier for developers to leverage AI to build their own Windows Apps.

Nadella went on to talk about Azure AI infrastructure. He stated that Microsoft has the "most comprehensive AI infrastructure in the world" and is expected to have data centers powered by 100% renewable energy by 2025. The number of countries where Microsoft's Azure cloud computing AI services are available has tripled.

He also mentioned that Microsoft will be one of the first cloud service providers to offer the latest Blackwell AI chip from NVIDIA, establishing a "deep partnership" with NVIDIA Microsoft CEO Nadella said that now more than 50,000 organizations are using Azure AI. With Azure AI, Microsoft provides the most extensive selection of cutting-edge and open-source models, including Large Language Models (LLM) and Small Language Models (SLM).

On Monday, Microsoft announced that OpenAI's latest flagship model, GPT-4o, will soon be integrated with Copilot. On Tuesday, Microsoft announced that the multimodal model GPT-4o, which integrates text, image, and audio processing, is now available in Azure AI Studio and can also be used as an API.

Nadella revealed at a developer conference that GPT-4o has been integrated into Azure AI, calling OpenAI's GPT-4o demo last week "very magical," and stating that OpenAI is Microsoft's "most strategic and important partner."

Microsoft stated that on the same day OpenAI announced GPT-4o, Microsoft opened the model for testing on Azure OpenAI services, and this Tuesday Microsoft announced full API access to GPT-4o.

On social media X, some netizens are excited about this development. A highly liked comment described it as a whole new world, greeting Copilot.

Some netizens are calling for the demonstration of native apps for writing Python code, where speaking and coding/copying chat content can be done simultaneously without the need for specific commands or key presses, just like having someone in the same room with you.

Azure AI's Phi-3-vision, a small multimodal model, is being tested on OPPO's new phone for Azure AI text-to-image conversion technology

Microsoft's AI SLM Phi-3 series released a new multimodal small model, Phi-3-vision, on Tuesday, which can be used in Azure.

Microsoft introduced that the Phi-3 model is powerful, cost-effective, and optimized for personal devices. Phi-3-vision has 4.2 billion parameters, can take in images and text, and provide text responses For example, users can ask questions about charts or open-ended questions about specific images.

Microsoft also revealed that Chinese smartphone manufacturer OPPO is piloting Azure AI's speech-to-text, rapid transcription, and Azure AI text-to-speech technologies on its new smartphones.

Team Copilot transforms personal assistants into team meeting hosts, chat collaborators, and project managers

Microsoft has launched various AI feature extensions for Copilot. One major extension announced on Tuesday is Team Copilot.

As the latest extension of the Copilot series of generative AI technologies, Team Copilot expands Copilot for Microsoft 365 from a behind-the-scenes personal AI assistant to a new important member of the team. You will be able to invoke Copilot in collaboration tools such as Teams, Loop, Planner, and more.

Team Copilot can act as a meeting host in meetings, manage meeting agendas, track meeting times, and record key meeting points; it can also serve as a collaborator in chats, providing important information, tracking action items, and resolving pending issues. It can also act as a project manager, helping ensure smooth progress on each project and promptly notifying the team for input. These initial experiences will be launched in preview later this year.

A comment on social media X mentioned that the extensions of Copilot look interesting, and Microsoft previewed some features obviously copied from ChatGPT: Builder UI, custom functions, GPT Mentions.

Copilot Studio Upgrade: Users can customize AI agents

Microsoft has upgraded Copilot Studio and introduced a brand new Agent feature, allowing developers to build Copilots that actively respond to data and events based on specific tasks and functions.

Copilots with these new features understand context, infer operations and inputs, learn based on user feedback, and seek help when unsure how to proceed, enabling them to independently manage complex, long-running business processes.

From IT equipment procurement to customer reception for sales and services, users can have Copilots built with Copilot Studio work for them. Before deployment, users can test the Copilot AI agent they designed and then use it across multiple channels

GitHub Copilot Extension Customization Enhances Copilot Experience

As one of the first major applications of Microsoft's generative AI, GitHub Copilot currently has 1.8 million paid users. This week, GitHub launched the first set of GitHub Copilot extensions developed by Microsoft and third-party partners, which has now entered an invite-only preview.

The GitHub Copilot extension is used to build an intelligent third-party ecosystem. Its new features allow developers and businesses to customize their GitHub Copilot experience directly in the GitHub Copilot Chat using their preferred services such as Azure, Docker, Sentry, and more.

GitHub Copilot for Azure demonstrates how to enhance development speed using natural language and broader functionalities. By using this extension through Copilot Chat, developers can explore and manage Azure resources, troubleshoot, find relevant logs, and code.

Edge Browser Enables Real-time Translation of YouTube, Bloomberg, and Other Website Videos in English, German, Italian, Spanish, and Hindi

Microsoft announced a new AI feature for its Edge browser on Tuesday, allowing users to receive real-time translation content through dubbing and subtitles while watching videos. Microsoft's Edge browser will soon provide real-time video translation on websites like YouTube, LinkedIn, Coursera, and more.

This feature will support translating Spanish to English, as well as translating English to German, Hindi, Italian, Russian, and Spanish. Edge will also support real-time translation of videos from news websites such as Reuters, CNBC, and Bloomberg. Microsoft plans to add more language translations in the future and expand the range of supported websites.

Media reviews suggest that besides providing a concise way to translate videos into the user's native language, Edge's new AI feature should make it easier for deaf or hearing-impaired users to watch videos.

Some netizens commented that if this real-time video translation feature is launched, it is definitely worth paying attention to. The question is, how accurate and fluent will the translations be, and how will the original speaker's tone and style be preserved.

The first preview version of Azure ND MI300X v5 virtual machine based on Arm Azure Cobalt officially released

In November last year, Microsoft released its first self-developed AI chip Azure Maia 100 and its first self-developed cloud processor Azure Cobalt 100. This Tuesday, Microsoft announced the launch of a preview version of the Cobalt 100 virtual machine (VM) customized based on the Arm architecture of Azure Cobalt 100. It is the first generation of virtual machines using Microsoft's new Cobalt processor, optimized for efficiency and performance when running general workloads and cloud-native workloads.

Nadella introduced that compared to similar Azure virtual machines, the Arm-based Cobalt 100 virtual machine can improve performance by up to 40%, making it the most efficient computing product launched by Microsoft Azure.

Microsoft's hardware systems and infrastructure leader Rani Borkar also told the media that preliminary tests showed that Azure Cobalt's performance is up to 40% higher than the performance of commercially available Arm servers currently used in their data centers. The media reported that Microsoft is currently testing the Cobalt CPU on workloads such as Microsoft Teams and SQL Server, planning to provide virtual machines for various workloads to customers next year.

The Maia 100 is currently being tested on GPT 3.5 Turbo, which also supports models such as ChatGPT, Bing AI workloads, and GitHub Copilot. Similar to the Cobalt CPU, Microsoft is currently in the early stages of deploying the Maia 100 and has not disclosed its specific specifications and performance.

Microsoft is the first cloud service provider to use AMD's MI300X AI accelerator chip for customer AI training and AI inference. This Tuesday, Microsoft announced that the Azure ND MI300X v5 virtual machine series has been officially released, optimized for high-demand AI and high-performance computing (HPC) workloads such as Azure OpenAI Service.

Real-time intelligence introduced in Microsoft Fabric data analysis platform

Microsoft's AI-supported data analysis platform, Microsoft Fabric, has also released new features. Users of this platform can utilize Real-Time Intelligence for instant decision-making, providing an end-to-end software as a service (SaaS) solution that allows customers to take action on massive, time-sensitive, high-granularity data to make business decisions faster and wiser Real-time Intelligence is now available for preview, not only helping data analysts to have a simple low-code/no-code experience, but also benefiting professional developers through a code-rich user interface.

Microsoft Fabric has launched a workload development kit, allowing Independent Software Vendors (ISVs) and developers to extend applications in Fabric, creating a unified user experience.

Microsoft Partners with Khan Academy to Provide AI Assistant Khanmigo for Free to U.S. K-12 Teachers

On Tuesday, Microsoft and Khan Academy announced a diverse partnership aimed at translating the transformative potential of AI into reality.

At the beginning of the collaboration, Microsoft will assist Khan Academy in providing Khanmigo for Teachers for free to all K-12 educators in the United States, equivalent to domestic primary and secondary education. This AI-driven educational assistant can free up teachers' time, allowing them to focus on the most important things - interacting with students and providing support. Microsoft will also donate Azure AI-optimized infrastructure permissions to enhance the availability of Khanmigo for Teachers. The service will be supported by Azure OpenAI Service.

Khan Academy will also collaborate with Microsoft to explore improving math tutoring in an economical, scalable, and adaptable way using the latest version of the SLM series Phi-3 developed by Microsoft. They also plan to integrate more Khan Academy teaching content into Copilot and Microsoft Teams Education Edition to provide learners with more resources