基于PaddleGAN快速让你的照片动起来
本项目基于PaddleGAN实现的FirstOrder与Wav2lip,。FirstOrder是输入一个模板视频与一张照片,就可以使照片里面的人物唱出模板视频里的歌曲,前段时间很火的 「蚂蚁呀嘿」就是用这个方法做的;还有另一个方法就是使用Wav2lip,输入照片和音频就可以直接让照片根据音频的内容动起来。
方案一:FirstOrder
1.下载PaddleGAN代码
In [ ]
# 从github上克隆PaddleGAN代码
!git clone https://gitee.com/paddlepaddle/PaddleGAN
In [ ]
# 安装所需安装包
%cd PaddleGAN/
!pip install -r requirements.txt
!pip install imageio-ffmpeg
%cd applications/
2.将驱动视频迁移至照片中
大家可以上传自己准备的视频和图片,并在下面的代码中source_image参数和driving_video参数分别换成自己的图片和视频路径,然后点击运行,就可以完成动作表情迁移,程序运行成功后,会在ouput文件夹生成名为result.mp4的视频文件,该文件即为动作迁移后的视频。
本项目中提供了原始图片和驱动视频供展示使用。具体的各参数使用说明如下
driving_video: 驱动视频,视频中人物的表情动作作为待迁移的对象
source_image: 原始图片,视频中人物的表情动作将迁移到该原始图片中的人物上
relative: 指示程序中使用视频和图片中人物关键点的相对坐标还是绝对坐标,建议使用相对坐标,若使用绝对坐标,会导致迁移后人物扭曲变形
adapt_scale: 根据关键点凸包自适应运动尺度
output:设置输出视频的存放文件夹
In [ ]
!export PYTHONPATH=$PYTHONPATH:/home/aistudio/PaddleGAN && python -u tools/first-order-demo.py --driving_video /home/aistudio/PlanA_FirstOrder/zhiming.mp4 \
--source_image /home/aistudio/PlanA_FirstOrder/trumps.png \
--relative \
--adapt_scale \
--output /home/aistudio/PlanA_FirstOrder/output/
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/setuptools/depends.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/layers/utils.py:26: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
def convert_to_list(value, n, name, dtype=np.int):
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/scipy/linalg/__init__.py:217: DeprecationWarning: The module numpy.dual is deprecated. Instead of using dual, use the functions directly from numpy or scipy.
from numpy.dual import register_func
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/scipy/special/orthogonal.py:81: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
from numpy import (exp, inf, pi, sqrt, floor, sin, cos, around, int,
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/scipy/io/matlab/mio5.py:98: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
from .mio5_utils import VarReader5
[04/08 21:00:32] ppgan INFO: Found /home/aistudio/.cache/ppgan/vox-cpk.pdparams
W0408 21:00:32.665690 1292 device_context.cc:362] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1
W0408 21:00:32.673117 1292 device_context.cc:372] device: 0, cuDNN Version: 7.6.
100%|█████████████████████████████████████████| 345/345 [00:12<00:00, 28.21it/s]
3.为生成的视频加上音乐
In [ ]
# add audio
!pip install moviepy
In [ ]
#为生成的视频加上音乐
from moviepy.editor import *
videoclip_driving = VideoFileClip("/home/aistudio/PlanA_FirstOrder/zhiming.mp4") # 打开驱动视频
videoclip_result = VideoFileClip("/home/aistudio/PlanA_FirstOrder/output/result.mp4") # 打开PaddleGan刚刚生成的视频
audio_driving = videoclip_driving.audio # 提取驱动视频里的声音
video = videoclip_result.set_audio(audio_driving) # 将提取出来的声音加到生成视频的音轨中
video.write_videofile("/home/aistudio/PlanA_FirstOrder/output/final_result.mp4", audio_codec="aac") # 保存视频
生成的视频将放置在“PlanA_FirstOrder/output/”目录下,下载后即可查看效果~
方案二:Wav2lip
(若运行了方案一,请先点击代码执行器—>重启执行器)
1.安装必要的资源包
In [1]
%cd /home/aistudio/work
# 安装所需安装包
!mkdir sndfile
%cd sndfile
!wget http://www.mega-nerd.com/libsndfile/files/libsndfile-1.0.28.tar.gz
!tar xzvf libsndfile-1.0.28.tar.gz
%cd libsndfile-1.0.28
!./configure --prefix=/home/aistudio/build_libs CFLAGS=-fPIC --enable-shared
!make
!make install
In [7]
%cd /home/aistudio/PaddleGAN
!pip install -r requirements.txt
%cd applications/
2.根据音频直接合成唇形动作
只需在如下命令中的face参数和audio参数分别换成自己的视频和音频路径,然后运行如下命令,就可以生成和音频同步的视频。
程序运行完成后,会在当前文件夹下生成文件名为outfile参数指定的视频文件,该文件即为和音频同步的视频文件。本项目中提供了demo展示所用到的视频和音频文件。具体的参数使用说明如下:
face: 原始视频,视频中的人物的唇形将根据音频进行唇形合成--通俗来说,想让谁说话
audio:驱动唇形合成的音频,视频中的人物将根据此音频进行唇形合成--通俗来说,想让这个人说什么
In [11]
!export PYTHONPATH=$PYTHONPATH:/home/aistudio/PaddleGAN && python tools/wav2lip.py --face /home/aistudio/PlanB_Wav2lip/MonaLisa.jpeg \
--audio /home/aistudio/PlanB_Wav2lip/TA.mp3 \
--outfile /home/aistudio/PlanB_Wav2lip/output/result.mp4
生成的视频将放置在“PlanB_Wav2lip/output/”目录下,下载后即可查看效果~