Can you also change the webpack publicPath to be that cdn?

That would be picked up by the html-webpack-plugin automatically.

make sure your value of "publicPath" property in "output" is your cdn address.

  output: {
    path: __dirname + '/dist',
    filename:[name].[contenthash:8].bundle.js',
    chunkFilename: [name].[contenthash:8].chunk.js', 
    publicPath: process.env.NODE_ENV === 'production' ?
      `your/cdn/address` :
      ''
  },

This issue had no activity for at least half a year. It's subject to automatic issue closing if there is no activity in the next 15 days.

see you [.use(HtmlWebpackPlugin)], guess you get error:
[ Error: Cannot call .tap() on a plugin that has not yet been defined. Call plugin('html').use(<Plugin>) first.].
if you configured multi-pages in vue.config.js, the [plugin('html')] not exist in config, coz it's replaced with the multi-pages settings, to identify the correct html plugin name, you need to run command to list all the plugins:

npx vue-cli-service inspect --plugins
[
  'vue-loader',
  'define',
  'case-sensitive-paths',
  'friendly-errors',
  'html-index',
  'html-demo',
  'preload-index',
  'prefetch-index',
  'preload-oeditor',
  'prefetch-oeditor',
  'copy',
  'ThemeColorReplacer',
  'IgnorePlugin'
]

Now you can use following to modify the plugin options

      config.plugin('html-index')
        .tap(args => {
          args[0].cdn = assetsCDN
          return args
        })
      config.plugin('html-demo')
        .tap(args => {
          args[0].cdn = assetsCDN
          return args
        })