Wear a mask when you go out, help your avatar P on N95 | Cloud Development Actual Combat

Wear a mask when you go out, help your avatar P on N95 | Cloud Development Actual Combat

Cherish life, start with me, hurry up and put on a mask, and introduce my open source Taro applet "Quickly Wear a Mask", which can intelligently recognize faces and put on a mask for group photos. (* ̄︶ ̄)

Using a  Taro cross-terminal framework, using Tencent's Yunyuan development model, and using face recognition based on Tencent Cloud's facial features analysis, the function of automatically wearing a mask for the avatar is realized.

Source address: https://github.com/shenghanqin/quickly-mask[1]

I am Sheng Hanqin , a front-end development engineer of Hujiang CCtalk, and an issue maintenance volunteer of the Taro framework. My  main focus is on front-end UI writing and team document construction.

The main function

  • Intelligent face recognition and facial features positioning
  • Support multi-person recognition
  • Support adding refueling pictures

Scan code preview

Search Wechat: Wear a mask quickly

Screenshot of Mini Program

Taro cloud development model

Taro is a set of multi-terminal development solutions that follow React's syntax specifications. Using Taro, we can write only one set of code, and then use Taro’s compilation tool to compile the source code separately and compile it on different ends (WeChat/Baidu/Alipay/ByteDance/QQ/JD Mini Program, KuaiApp, H5, React-Native, etc.) running code.

Originally, my plan was a small program to cooperate with the API service built on Tencent Cloud personal server  nodejs +  expressimplementation. It's just that the API request I personally configured is not ideal, because the intermediate path from the applet to the Tencent Cloud personal server to the Tencent Cloud service is relatively long.

At this point, I discovered that Taro has integrated the Tencent Cloud development model. It took more than an hour to switch from the applet + personal server to the Tencent Cloud development model (after getting familiar with cloud development, configuring the cloud development environment and other small details).

//taro/src/pages/wear-a-mask
Taro.cloud.callFunction({
  name:'analyze-face',
  data: {fileID: '12345'}
}).then(res => console.log(res))

//cloud/functions/analyze-face Tencent Cloud face recognition effect
return new Promise((resolve, reject) => {
 //To call the interface you want to access through the client object, you need to pass in the request object and the response callback function
  client.AnalyzeFace(faceReq, function (error, response) {
   //Request exception return, print exception information
    if (error) {
      const {code =''} = error

      resolve({
        data: {},
        time: new Date(),
        status: -10086,
        message: status.FACE_CODE[code] || '图片解析失败'
      })
      return
    }
    console.log('AnalyzeFace response :', response)

   //请求正常返回,打印response对象
    resolve({
      data: response,
      time: new Date(),
      status: 0,
      message: ''
    })
  })
});

腾讯云五官识别

其实,启发我做这个小程序的是这两个文章,《「圣诞特辑」纯前端实现人脸识别自动佩戴圣诞帽[2]》和《我要戴口罩 – 为微信、微博等社交网络头像戴口罩[3]》。

Because of the spread of the novel coronavirus, wearing a mask is a necessary preventive measure. So how can we innovate? During the process of using the "I want to wear a mask" applet, I found that the position of the mask is manually moved, so I want to automatically wear it, just like the "automatic identification of wearing a Christmas hat" I saw earlier. , Then I’ll just wear a mask.

In "Automatically wear Christmas hats", the solution used is a pure front-end face-api. If you want to put it in a small program, you will have the following small problems:

  • The face-api recognition model is more than 5M in size, even if it is loaded purely by the front-end, it appears to be relatively large. However, there is still a difference between the canvas of the applet and the web page, and the face-api cannot be used directly.
  • Face-api is loaded on nodejs, and it needs cooperation tensorflowand canvassimulation. After the actual implementation, it is found that the image recognition process is still relatively slow (after the image is uploaded, the content of the image is obtained, the position of the facial features is recognized, and the facial features are returned), it is easy to cause the interface request to time out.

In the process of using Tencent Cloud, I found that Tencent Cloud’s artificial intelligence category actually has a human face recognition function. After careful analysis , I found that there is " Facial Analysis [4] ". The returned data and the face-apireturned data format are still Very similar, the monthly free quota of "Face Recognition" was 10,000 times, which made me very happy at the time.

Of course, a very big pitfall in the use process is that my implementation process requires uploading pictures with a size of 1M or more, and the "Facial Analysis" signature method is required TC3-HMAC-SHA256. The official npm version tencentcloud-sdk-nodejsdoes not support this signature method. You need to download it from the official GitHub[ 5]signature3 Download the corresponding code on the branch of the library .

Crop the picture

Another pain point in the "I want to wear a mask" applet is that if you upload a rectangular picture, it will be forcibly turned into a square. I was thinking about how to crop a square image. At this time, I found taro-cropperthis powerful image cropping plugin in the npmjs warehouse (also available in the Taro material market).

Mask positioning

After deriving the facial features of the face from the "Facial Analysis", how to put a mask on the face based on this? "Automatic recognition of wearing Christmas hats" is based on the "three courts and five eyes" to calculate the position of the Christmas hats. For masks, it is actually simpler to get the midpoint of the mouth, and then rotate and zoom.

  • To get the mouth information, please read the " Christmas Special" Pure Front End Realization of Face Recognition and Automatic Wearing of Christmas Hats[6] , and then fancy taro/src/utils/face-utils.jsthe getMouthInfomethod.
  • For mask positioning, you can see the code below~
let shapeList = info.map(item => {
  let {faceWidth, angle, mouthMidPoint, ImageWidth} = item
  let dpr = ImageWidth/CANVAS_SIZE * (375/windowWidth)
  const maskCenterX = mouthMidPoint.X/dpr
  const maskCenterY = mouthMidPoint.Y/dpr
  const scale = faceWidth/MASK_SIZE/dpr
  const rotate = angle/Math.PI * 180

  return {
    maskSize: DEFAULT_MASK_SIZE,
    maskCenterX,
    maskCenterY,
    scale,
    rotate,
  }
})

Mask positioning

  • When touchstart, save the touch starting point at this time, and use the base map and mask position at this time as the reference point for calculation of the rotation angle and zoom ratio value
  • When touchmove, calculate the moving distance in the x/y direction according to the starting point and the temporary ending point, calculate the reference point and add this distance separately to get the moved position, and calculate the change of the position before and after the movement. Rotation angle and zoom ratio
  • When touchend, reset the position, rotation angle and zoom ratio of the basemap and mask

Canvas drawing

  • First draw the base map (calculate the coordinates of the upper left and lower right corners according to the screen size and picture size)
  • Draw the mask (calculate the size of the final mask and the rotation angle of the center position, move the origin of the canvas to the center position of the mask, rotate the canvas and draw the mask)

In the WeChat applet, canvas drawing needs to change the network picture into a local picture. If you download it while drawing, there is a risk of a long download time and easy download failure. It is more reliable to use a local picture. In the Tarojs project, the canvas drawing drawImage(src, 0, 0, 300, 300)method, if src is set require('../../images/xxx.png'), you must remember to modify the following configuration, otherwise the applet will report an http  500error.

{
  mini: {
    imageUrlLoaderOption: {
      limit: 0
    }
}

Reference project

  • Mini Program: I want to wear a mask, idealclover/Wear-A-Mask[7]
  • Automatic Christmas hat: christmas-hat[8]
  • Example of automatic face recognition : bnk48-face-recognition[9]
  • Mini Program: Christmas Hat : jasscia/ChristmasHat[10]
  • face-api.js  justadudewhohacks/face-api.js[11]
  • Tencent Cloud Face Recognition:  https://cloud.tencent.com/product/facerecognition[12]
  • Tarojs version picture cropping: SunnyQjm/taro-cropper[13]
  • Virus evolution simulator: qqqdu/zhonghuajia[14]

Reference

[1]https://github.com/shenghanqin/goddess-hat:  https://github.com/shenghanqin/quickly-mask

[2] "Christmas Special" pure front end realizes face recognition and automatically wears Christmas hats:  https://juejin.im/post/5e02b73fe51d455807699b1f

[3] I want to wear a mask – wear a mask for social network avatars such as WeChat and Weibo:  https://www.appinn.com/woyaodaikouzhao-wechat-miniapp/

[4] 5.senses analysis:  https://cloud.tencent.com/document/api/867/32779

[5]Official GitHub:  https://github.com/TencentCloud/tencentcloud-sdk-nodejs/tree/signature3

[6] "Christmas Special" pure front end realizes face recognition and automatically wears Christmas hats:  https://juejin.im/post/5e02b73fe51d455807699b1f

[7]idealclover/Wear-A-Mask:  https://github.com/idealclover/Wear-A-Mask

[8]christmas-hat:  https://github.com/hk029/christmas-hat

[9]bnk48-face-recognition:  http://supachaic.github.io/bnk48-face-recognition

[10]jasscia/ChristmasHat:  https://github.com/jasscia/ChristmasHat

[11]justadudewhohacks/face-api.js:  https://github.com/justadudewhohacks/face-api.js

[12]https://cloud.tencent.com/product/facerecognition:  https://cloud.tencent.com/product/facerecognition

[13]SunnyQjm/taro-cropper:  https://github.com/SunnyQjm/taro-cropper

[14]qqqdu/zhonghuajia:  https://github.com/qqqdu/zhonghuajia

Cloud Development (CloudBase) is a cloud-integrated product solution. It adopts a serverless architecture, avoids operation and maintenance such as environment construction, supports one cloud and multiple terminals, and helps to quickly build small programs, web applications, and mobile applications.

Technical documentation: https://www.cloudbase.net/

WeChat search: Tencent cloud development, get the latest progress of the project

Reference: https://cloud.tencent.com/developer/article/1607520 Wear a mask when you go out, and help your avatar P on N95 | Cloud Development Practice-Cloud + Community-Tencent Cloud