How Meta introduced AV1 to Reels

• We’re sharing how we’re enabling manufacturing and supply of AV1 for Fb Reels and Instagram Reels.
• We imagine AV1 is probably the most viable codec for Meta for the approaching years. It affords larger high quality at a a lot decrease bit fee in contrast with earlier generations of video codecs.
• Meta has labored intently with the open supply group to optimize AV1 software program encoder and decoder implementations for real-world, global-scale deployment.

As individuals create, share, and devour an ever-increasing quantity of on-line movies, Meta is working to develop probably the most bandwidth-efficient methods to transcode content material whereas sustaining affordable compute and energy consumption ranges. Selecting probably the most acceptable video coding codecs — the algorithms for compressing and decompressing the file — is essential. Over the previous 20 years, researchers have developed video coding requirements with ever-higher compression effectivity, together with AVC, HEVC, and VVC, developed by MPEG/JVET, and VP9 and AV1, developed by Google and the Alliance for Open Media (AOM). Newer-generation normal usually can cut back the bandwidth by about 30 p.c to 50 p.c in contrast with its predecessor whereas sustaining comparable visible high quality. On the identical time, nevertheless, every new normal has consumed considerably extra power and compute than the final, whereas necessitating encoders that have been many instances extra complicated.

We imagine AV1 would be the most viable codec for Meta over the following a number of years. AV1 is the first-generation royalty-free video coding normal developed by AOM, of which Meta is a founding member. It delivers about 30 p.c higher coding effectivity than VP9 and HEVC — permitting individuals who use our apps to take pleasure in high-quality video at a lot decrease bandwidth, and enabling us to maximise storage effectivity and cut back egress site visitors, CDN prefetching/caching, and community congestion. AV1 additionally has a a lot richer characteristic set than different video coding requirements and might help most of Meta’s typical manufacturing usages. AV1 is royalty-free, and each the encoder and decoder implementations are open sourced, with very energetic growth and good help.

Over the previous few years, Meta has labored intently with the open supply group to optimize AV1 software program encoder and decoder implementations for real-world, global-scale deployment. Our objective is to enhance playback from what we presently supply with AVC and VP9. We need to make sure that as we roll out AV1, it delivers actual worth to the individuals who use our apps.

Discovering the best AV1 encoders and decoders

A number of open supply and closed-source encoder implementations are prepared for manufacturing, all virtually as environment friendly because the AV1 reference encoder. In a paper, “Towards much better SVT-AV1 quality-cycles tradeoffs for VOD applications,” collectively printed with Intel eventually 12 months’s SPIE convention, we benchmarked a number of open supply encoders — together with x264, x265, libvp9, libaom, SVT AV1, and VVC reference encoder (vvenc) — for a video on demand (VOD) use case. The graph beneath illustrates the trade-off between encoder high quality (vertical axis) and complexity (horizontal axis). Each level on the graph corresponds to an encoder preset. The y-axis represents the common BD-rate relative to libaom cpu-used=0; decrease values point out higher coding effectivity. The x-axis represents the encoding time in seconds in logarithmic scale.

Just a few highlights from this graph:

• SVT-AV1, the productization encoder for the AV1 coding normal, maintains constant efficiency throughout a variety of complexity ranges. With a complete of 13 presets, SVT-AV1 can cowl a complexity vary that extends from the upper high quality AV1 to the upper speeds AVC presets similar to greater than 1000x change in complexity. This complexity vary covers all open supply software program encoders utilized in manufacturing techniques.
• At any given level on the x-axis, SVT-AV1 can maximize coding effectivity in contrast with some other manufacturing encoder. For instance, the M12 preset has comparable complexity efficiency to the x264 veryfast preset, however M12 is about 30 p.c extra environment friendly.
• At any given level on the y-axis, SVT-AV1 can maximize encoding velocity in contrast with some other manufacturing encoder. For instance, the M8 preset is about as environment friendly as libvp9 preset 0, however M8 is sort of 10 instances quicker.

SVT-AV1 affords 13 presets, permitting a fine-grained trade-off between high quality and velocity. Extra importantly, SVT-AV1 now features a “-fast-decode” choice, which accelerates software program decoding — with solely a slight drop in effectivity — by routinely limiting or disabling using AV1 coding instruments that aren’t software-decoder pleasant. SVT-AV1 additionally offers thread administration parameters to stability density and velocity — vital for large-scale manufacturing — doubtlessly enabling a one- or two-second delay for dwell video streaming. Many parameters could be adjusted to enhance coding effectivity or to help sure manufacturing situations. Some AV1 coding instruments that have been proposed to be used circumstances in deployment, reminiscent of reference body scaling, tremendous decision, movie grain synthesis, and swap frames, are additionally supported in SVT-AV1.

Our largest problem can be client-side decoding of AV1. Many {hardware} distributors, together with Intel and NVIDIA, have begun to help AV1 {hardware} decoding on PC. Nonetheless, we’re serving video primarily to cell phones, most of which don’t embody AV1 {hardware} decoders. For now, we should rely totally on software program decoders. Two main open supply software program decoders are appropriate with a number of platforms: dav1d was developed by VideoLAN and the open supply group and might function an app-level decoder, whereas Google’s libgav1 is built-in into the Android SDK.

After extensively benchmarking the decoders’ efficiency, specializing in sides reminiscent of useful resource necessities, crashes and responsiveness, and body drops, we determined to combine dav1d into the participant for each iOS and Android platforms. Now we have been working intently with the open supply group to optimize dav1d’s efficiency. Within the final 12 months, we additionally labored with Ittiam to conduct a benchmark take a look at on Android telephones. dav1d can help 720p30 real-time playback on a lot of the gadgets in our pattern, attaining 1080p30 on sure mid-range and high-end fashions.

Some Android telephones, such because the Google Pixel 6 Professional and Samsung Galaxy S21, already help {hardware} AV1 decoding. Within the close to future, we anticipate {that a} rising variety of high-end Android fashions will help AV1 {hardware} decoding, with mid-tier gadgets following finally.

Deploying AV1 encoding on Fb Reels and Instagram Reels

Early in 2022, we deployed AV1 encoding for Fb and Instagram Reels. When somebody uploads a video, the platform generates a number of bit-rate encodings tailor-made to the video’s projected watch time. To forestall stalling brought on by modifications in bandwidth, purchasers can choose the model that most closely fits their connection velocity — a method referred to as adaptive bit fee (ABR) streaming. For movies with excessive projected watch time, we use superior ABR encoding based mostly on the convex hull dynamic optimizer algorithm. For every uploaded video, we produce a number of down-scaled variations and encode every with a number of quantization parameters (QPs) and Fixed Charge Elements (CRFs). For instance, for a 1080p video, we would create seven resolutions and 5 CRFs, for a complete of 35 encodings. After encoding, the system upscales decoded movies to the unique decision and calculates the standard rating.

Within the graph of fee distortion (RD) curves beneath, the x-axis represents the encoding bit fee and the y-axis the standard rating, expressed in FB-MOS items on a scale of 0 to 100.

From these 35 RD factors, we calculate the convex hull, a curve that connects the RD factors on the higher left boundary. (Theoretically, if we may use all potential encoding resolutions and CRFs to supply a a lot denser plot, any level on the convex hull would be the most optimum encoding choice for this video when it comes to decision and CRF worth.) As illustrated above, we are able to then choose one of the best encoding for supply based mostly on the goal high quality or bit fee.

Now we have simplified this difficult course of. In earlier research, we discovered that we may use the high-speed preset for first-pass encoding and to supply the convex hull, after which take a second cross to encode the chosen (decision, CRF) factors with the high-quality preset. Although this strategy requires further encoding, it’s quicker as a result of the primary cross could be completed far more shortly. (Coding effectivity drops solely barely.) This strategy works even when the primary and second passes use completely different encoders. For instance, we are able to use AVC or VP9 within the first cross and AV1 within the second. We are able to additionally leverage the {hardware} encoder in our internally designed ASICs to speed up this course of.

In the long run, we selected a two-stage hybrid {hardware}/software program ABR encoding strategy. {Hardware} AVC encoding is triggered at video add time; for this stage, we retailer solely the standard and bit fee data however not encoded bitstreams. When projected watch time of the video exceeds the brink, second stage encoding is triggered with software program AVC, VP9 or AV1 encoder based mostly on the chosen (decision, CRF) on the convex hull.

We are able to simply add AV1 as one of many second-stage encoders; it’s already deployed for Fb Reels. Now we have applied the same heuristic-based strategy for Instagram Reels. For one instance video proven within the graph above, three encoding households with AVC, VP9, and AV1 have been produced. Their RD curves intently observe the convex hull from the first-stage encoding. For this specific video instance,  the best-quality AV1 encoding rivals these of the opposite two requirements, however with a bit fee 65 p.c decrease than AVC’s and 48 p.c decrease than VP9’s. As well as, AV1 achieves the specified high quality inside a really slender bit fee vary, so we are able to additional cut back compute and storage prices by producing fewer encodings throughout the second stage. Consequently, individuals who use our merchandise can take pleasure in high-quality video at a lot decrease bandwidth.

AV1 decoder integration and testing

It was comparatively simple to allow AV1 decoding and playback on the iOS gadgets. After just some rounds of assessments, we began supply. To combine the dav1d decoder on iOS, we discovered that two to 4 threads would meet most of our manufacturing wants; any further threads would waste reminiscence and energy with out boosting efficiency.

dav1d has two modes: synchronous and asynchronous. In synchronous mode, dav1d decodes one body at a time however permits low-latency decoding for every body. In asynchronous mode, dav1d decodes a number of compressed frames in parallel, suspending rendering till all frames are decoded. In principle, asynchronous mode offers larger throughput and quicker decoding. For now, we undertake synchronous mode on iOS because it matches the present participant stack, however we’re trying into migrating to asynchronous mode sooner or later.

To help the decoding of 10-bit AV1-encoded HDR video, we constructed a single dav1d binary that helps each 8- and 10-bit decoding and ensures that coloration data is preserved within the transcoding course of.

The Android platform offered greater challenges. First, as a result of individuals have interaction with our apps on an unlimited variety of Android fashions, we needed to run native and large-scale A/B assessments on numerous gadgets to search out the optimum decoder configurations. To assist debug and triage issues from the AV1 decoder library, we added intensive logging that propagated again error messages from all through the participant stack. This vital step helped us shortly establish and resolve points within the integration course of.

Second, as a result of we’re utilizing app degree software program decoders, we used the {hardware} VP9 decoder and software program AV1 decoder collectively when taking part in the identical video stream, to appropriately help blended codec manifest and in-stream ABR lane swap. We would have liked to verify they interacted with the render engine appropriately.

We additionally wanted to help gadgets with low efficiency and show decision. (This was not an issue with iPhones.) Though AV1 can encode high-resolution movies at a a lot decrease bit fee than VP9, bit fee discount is smaller for low-resolution movies. That makes it troublesome to point out enchancment in top-line supply metrics for low-performance Android telephones. We responded through the use of higher-quality encoding presets to spice up coding effectivity in low-resolution ABR lanes.

One other problem was that reminiscence allocation and thread creation elevated the decoding latency of the primary few video frames, prolonging the software program decoder begin time, delaying participant startup, and inflicting in-play stalls. This was most difficult with Reels, as a result of individuals usually scroll throughout a number of Reels movies in fast succession. To enhance scrolling efficiency, we prefetched a number of Reels movies earlier, earlier than they have been performed.

Earlier than we conduct a large-scale A/B supply take a look at, we have now to test whether or not the tip system is highly effective sufficient for real-time decoding and playback of AV1 bitstreams. Nonetheless, there isn’t a simple solution to classify Android telephone efficiency. We can’t take a look at each mannequin that exists, as there are literally thousands of them. And traits reminiscent of core counts, chipset distributors, RAM measurement, and 12 months and mannequin should not enough indicators of functionality. We finally determined to run a small benchmarking take a look at to measure efficiency and provides every telephone a efficiency rating. This benchmarking take a look at consisted of primary compute operations, together with Gaussian blur, reminiscence allocation, reminiscence copy, and 3D rendering. With this strategy, we may assign scores to any current or upcoming cell phones and group them based mostly on these numbers. Our A/B assessments then recognized the fashions that would help 720p, 1080p, and 10-bit HDR playback.

After the preliminary Android rollout, we began to allow AV1 {hardware} decoding for the few Android telephones that help it. We anticipate {hardware} decoding to enhance AV1 efficiency, and we plan to carry out large-scale assessments when a bigger variety of succesful telephones change into accessible.

Newest supply standing

We began the AV1 supply for Fb Reels on iPhone in early 2022 and noticed the advantages inside the first week of the rollout. 

The next graph exhibits the week-over-week common playback FB-MOS for all Fb Reels movies performed on iPhones. Playback FB-MOS improved by about 0.6 factors after we deployed AV1. 

This second graph exhibits the common bit fee for all Fb Reels movies performed on iPhones. AV1 lowered the common bit fee by 12 p.c. 

This final graph exhibits the watch time of various codecs for Fb Reels on iPhone. AV1 watch time rose to about 70 p.c throughout the first week of rollout.

Now we have continued to allow new options for iPhone, together with 1080p30 8-bit AV1 supply for iPhone 8 and past, 10-bit HDR supply as much as 1080p30 for fashions of iPhone X and past that help HDR show, and 1080p60 8-bit AV1 supply for iPhone 11 and past. AV1 encodes a excessive share of the Fb Reels and Instagram Reels movies watched on iPhones. Now we have additionally enabled 8-bit AV1 supply to pick midrange to high-end Android telephones. The watch time share on Android for AV1 is comparatively small however rising.

What’s subsequent for AV1 at Meta?

AV1 delivers actual worth to the individuals who use our merchandise. It affords larger high quality at a a lot decrease bit fee in contrast with earlier generations of video codecs. For instance, within the video beneath, there may be an apparent distinction in high quality between AVC, VP9, and AV1 at roughly the identical bit fee.

Going ahead, we are going to proceed to develop AV1 supply for Android telephones and allow {hardware} decoding in new gadgets that help it.

For low-end Android telephones, it stays difficult to play again high-resolution AV1 bitstreams. To deal with this, we’re presently experimenting with blended codec manifest help. On the server facet, the ABR supply algorithm generates a blended codec manifest that incorporates a number of video adaptation units with bitstreams encoded utilizing completely different codecs, reminiscent of VP9 and AV1. It additionally specifies which AV1 and VP9 lanes the system ought to select from based mostly on its efficiency rating. For instance, a low-end telephone can play AV1 as much as 540p and swap to VP9 for larger decision lanes. 

With increasingly more {hardware} distributors implementing AV1 decoders in cell SOCs, we anticipate the variety of AV1 succesful gadgets to proceed to develop within the subsequent few years, permitting extra finish customers to take pleasure in the advantages of AV1.

Acknowledgements

This work is a collective effort by the Video Infra workforce and Instagram workforce at Meta, together with exterior companions, together with the Intel SVT workforce, VideoLAN, Ittiam, Two Orioles, and the open supply group. The authors want to thank Jamie Chen, Syed Emran, Xinyu Jin, Ioannis Katsavounidis, Denise Noyes, Mohanish Penta, Nam Pham, Srinath Reddy, Shankar Regunathan, David Ronca, Zafar Shahid, Nidhi Singh, Yassir Solomah, Cosmin Stejerean, Wai Lun Tam, Hassene Tmar, and Haixiong Wang for his or her contributions and help.