Skip to content

Auto scaling not only by request/seconds but also considering CPU/Memory usage #379

New issue
New issue
Closed
Closed
Auto scaling not only by request/seconds but also considering CPU/Memory usage#379
Labels
proposalquestion
@rayhero

Description

@rayhero
rayhero
opened on Nov 9, 2017

Currently the auto scaling is base on request time/sec. Can we trigger the auto scaling by CPU/Memory usage? Currently if the request less than 5 times/secs and the cpu/memory resource is out of limits, the function will be terminated.

Expected Behavior

The auto scaling is not only trigger by request/secs but also trigger by replica sets's CPU & Memory's usage

Current Behaviour

Currently if the request less than 5 times/secs and the cpu/memory resource is out of limits, the function will be terminated.

Possible Solution

Base on alex's demo video. Maybe add the auto scaling rule on deploy.go and prometheus to alert
the resource full.

Steps to Reproduce (for bugs)

  1. deploy a high resource needed function on openfaas
  2. trigger job less than 5 times/second but can exceed pod's CPU/memory
  3. the function will be terminate and not trigger auto scaling

Context

Since I want to dynamic scaling my replica sets to provide different amount request. But my
bottle neck is not the request time/sec. My function need large CPU/Memory. I can manual
create pods to make my replica can afford more function request. but when the replica set
out of resource. the function request will loss.

Your Environment

  • Docker version docker version (e.g. Docker 17.0.05 ):
    Client:
    Version: 17.09.0-ce
    API version: 1.32
    Go version: go1.8.3
    Git commit: afdb6d4
    Built: Tue Sep 26 22:40:09 2017
    OS/Arch: darwin/amd64
    Server:
    Version: 17.09.0-ce
    API version: 1.32 (minimum version 1.12)
    Go version: go1.8.3
    Git commit: afdb6d4
    Built: Tue Sep 26 22:45:38 2017
    OS/Arch: linux/amd64
    Experimental: true
  • Are you using Docker Swarm or Kubernetes (FaaS-netes)?
    Kubernetes (FaaS-netes)
  • Operating System and version (e.g. Linux, Windows, MacOS):
    Local: MacOS 10.12.5, remote: GCP k8s
  • Link to your project or a code example to reproduce issue:

Activity

open-derek
added
proposal
on Nov 9, 2017
alexellis

alexellis commented on Nov 11, 2017

@alexellis
alexellis
on Nov 11, 2017
Member

Entirely valid scenario, here's two ideas:

1. Change alerts/metrics

  • Install NodeExporter and capture metrics on each host / container
  • Update the alerts

2. Use K8s built-in scaling

  • Add a regular Kubernetes pod scaler policy

There may be other suggestions worth exploring too. Option 1 is not going to work with Swarm, Nomad, Hyper, etc.

rayhero

rayhero commented on Nov 13, 2017

@rayhero
rayhero
on Nov 13, 2017
Author

@alexellis thanks for the suggestion! I'll try to integrate the NodeExporter in prometheus.

rayhero

rayhero commented on Nov 13, 2017

@rayhero
rayhero
on Nov 13, 2017
Author

@alexellis : I try to use kubectl create -f node-exporter.yaml to add node-exporter on kubernetes. But the prometheus can not find the node-exporter on target page. The yaml I using: https://github.com/cedriclam/example-Prometheus-with-Kubernetes-and-Grafana/blob/master/node-exporter.yaml
I try to set the namespace but still not work. How to add node-exporter in openfaas?

rayhero

rayhero commented on Nov 13, 2017

@rayhero
rayhero
on Nov 13, 2017 · edited by rayhero
Author

I try to use the sample (https://github.com/cedriclam/example-Prometheus-with-Kubernetes-and-Grafana) to install nodeExporter on Kubernetes. And I add the scrape setting on faas-netes's configmaps.yaml:


- job_name: 'kubernetes-pods'
  kubernetes_sd_configs:
  - api_servers:
    - 'https://kubernetes.default.svc.cluster.local'
    in_cluster: true
    role: pod

  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_namespace,
__meta_kubernetes_pod_label_name]
    separator: '/'
    target_label: job
  - source_labels: [__meta_kubernetes_pod_node_name]
    target_label: node

The prometheus show the nodeExpoter info on target page. But the status is down. How to make prometheus can get info from nodeExporter? Does I still miss some settings?

2017-11-13 4 33 18

rayhero

rayhero commented on Nov 13, 2017

@rayhero
rayhero
on Nov 13, 2017
Author

I fix the bug. It's the network issue. The fix flow is install nodeExporter first. And set nodeExporter's ip in scrape's settings. I can got the node info from nodeExporter now lol.

rayhero

rayhero commented on Nov 15, 2017

@rayhero
rayhero
on Nov 15, 2017
Author

I can success fire the alert on prometheus. But I can not see the pod number increase. How do control the auto scaling in openfaas? I check to code and found the control flow is in configmaps,
so I add the node resource monitoring:

scrape_configs:
      # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
      - job_name: 'prometheus'

        # Override the global default and scrape targets from this job every 5 seconds.
        scrape_interval: 5s

        # metrics_path defaults to '/metrics'
        # scheme defaults to 'http'.
        static_configs:
          - targets: ['localhost:9090']

      - job_name: 'gateway'
        scrape_interval: 5s
        dns_sd_configs:
          - names: ['gateway.{{ .Release.Namespace }}']
            port: 8080
            type: A
            refresh_interval: 5s

      # 20171113 ray: need manual set nodeExporter ip
      - job_name: 'node resources'
        scrape_interval: 5s
        static_configs:
          - targets:
            - '10.55.248.65:9100'
        params:
          collect[]:
            - cpu
            - meminfo
            - diskstats
            - netdev
            - netstat

      # 20171113 ray: need manual set nodeExporter ip
      - job_name: 'node storage'
        scrape_interval: 1m
        static_configs:
          - targets:
            - '10.55.248.65:9100'
        params:
          collect[]:
            - filefd
            - filesystem
            - xfs

then I notice that I need to set the alert rule to observe resource:

alert.rules: |
    ALERT service_down
      IF up == 0

    ALERT APIHighInvocationRate
      IF sum ( rate(gateway_function_invocation_total{code="200"}[10s]) ) by (function_name) > 5
      FOR 5s
      LABELS {
        service = "gateway",
        severity = "major",
        value = "{{ "{{" }}$value{{ "}}" }}"
      }
      ANNOTATIONS {
        summary = "High invocation total on {{ "{{" }} $labels.instance {{ "}}" }}",
        description =  "High invocation total on {{ "{{" }} $labels.instance {{ "}}" }}"
      }

    ALERT NodeCPUUsage
      IF (100 - (avg by (instance) (irate(node_cpu{job="node resources",mode="idle"}[5m])) * 100)) > 25
      FOR 5s
      LABELS {
        service = "gateway",
        severity = "major",
        value = "{{ "{{" }}$value{{ "}}" }}"
      }
      ANNOTATIONS {
        summary = "{{ "{{" }} $labels.instance {{ "}}" }}: High CPU usage detected",
        description =  "{{ "{{" }} $labels.instance {{ "}}" }}: CPU usage is above 80% (current value is: {{ "{{" }} $value {{ "}}" }})"
      }

    ALERT NodeMemoryUsage
      IF (((node_memory_MemTotal-node_memory_MemFree-node_memory_Cached)/(node_memory_MemTotal)*100)) > 25
      FOR 5s
      LABELS {
        service = "gateway",
        severity = "major",
        value = "{{ "{{" }}$value{{ "}}" }}"
      }
      ANNOTATIONS {
        summary = "{{ "{{" }} $labels.instance {{ "}}" }}: High memory usage detected",
        description =  "{{ "{{" }} $labels.instance {{ "}}" }}: Memory usage is above 80% (current value is: {{ "{{" }} $value {{ "}}" }})"
      }

I use the same label which the autoscaling example used. I can see the alert firing on the Prometheus alert. But I don't see the pod increase like echo example.

If I use kubectl autoscale deployment auto-hotspot --cpu-percent=20 --min=1 --max=100 --namespace=openfaas-fn

It can success auto scaling the pod number. Can some one teach me where to control the auto scaling in fass-netes?

alexellis
added
question
on Nov 15, 2017
alexellis
mentioned this in 2 issues on Nov 25, 2017
kaikai-liu

kaikai-liu commented on May 17, 2018

@kaikai-liu
kaikai-liu
on May 17, 2018 · edited by kaikai-liu

@rayhero Hi Rayhero, I have some clues for your concerns.

I configured prometheus to scrape the cAdvisor metrics in Kubelet for every pod and created related CPU usage alerts. I can see that the alert is firing when the CPU usage of a pod in the namespace of openfaas-fn exceeds the alert value. But that does not trigger Kubernetes or OpenFaaS gateway to create more pods. The reason I think is the following.

The gateway alert is from gateway invocation rate and the labels of the alert is like {alertname="APIHighInvocationRate" function_name="find-prime" service="gateway" severity="major" value="18"}; but the CPU usage alert is from container_cpu_usage_seconds_total and the alert label is like {alertname="APIHighCPUUsage" pod_name="find-prime-79659f57bd-cxkkj" service="gateway" severity="major" value=0.47"}. The active alerts that are firing are sending notifications to the OpenFaaS gateway so that gateway may scale up the pods. But the gateway may not recognize the latter alert with a label of pod_name="find-prime-79659f57bd-cxkkj" and, therefore, it does not scale up the pods.

And it turned out that my guess is right. After I added a lable of function_name="find-prime" to the CPU usage alert, the gateway started to scale up the pod!

alexellis

alexellis commented on Aug 7, 2018

@alexellis
alexellis
on Aug 7, 2018
Member

Derek close: inactive

derek
closed this as completedon Aug 7, 2018
alexellis

alexellis commented on Aug 7, 2018

@alexellis
alexellis
on Aug 7, 2018
Member

That sounds really useful @kaikai-liu - would you be wiling to write-up the steps here?

kaikai-liu

kaikai-liu commented on Aug 7, 2018

@kaikai-liu
kaikai-liu
on Aug 7, 2018

@alexellis I added more scrape setting of kubelet cAdvisor and more alert rule setting in prometheus config file and then I can see the pod scales up when there is a firing alert in prometheus. The following is the revised version of the yaml file prometheus_config.yml.

kind: ConfigMap
apiVersion: v1
metadata:
  labels:
    app: prometheus
  name: prometheus-config
  namespace: openfaas
data:
  prometheus.yml: |
    global:
      scrape_interval:     15s
      evaluation_interval: 15s
      external_labels:
          monitor: 'faas-monitor'
    rule_files:
        - 'alert.rules.yml'
    scrape_configs:
      - job_name: 'prometheus'
        scrape_interval: 5s
        static_configs:
          - targets: ['localhost:9090']
      - job_name: "gateway"
        scrape_interval: 5s
        dns_sd_configs:
          - names: ['gateway.openfaas']
            port: 8080
            type: A
            refresh_interval: 5s
      - job_name: "cadvisor"
        scrape_interval: 5s
        metrics_path : '/metrics/cadvisor'
        static_configs:
          - targets:
              - '10.239.85.140:10255'
            labels:
              node: lkk-skx
          - targets:
              - '10.239.66.149:10255'
            labels:
              node: lkk-nuc
          - targets:
              - '10.239.85.157:10255'
            labels:
              node: lkk-nuc1
        metric_relabel_configs:
        - source_labels: ['pod_name']
          regex: 'find-prime-79c698cd7b-4f67x'
          replacement: 'find-prime'
          target_label: 'function_name'
    alerting:
      alertmanagers:
      - static_configs:
        - targets:
          - alertmanager.openfaas:9093
  alert.rules.yml: |
    groups:
    - name: openfaas
      rules:
      - alert: service_down
        expr: up == 0
      - alert: APIHighInvocationRate
        expr: sum(rate(gateway_function_invocation_total{code="200"}[10s])) BY (function_name)
          > 5
        for: 5s
        labels:
          service: gateway
          severity: major
          value: '{{$value}}'
        annotations:
          description: High invocation total on {{ $labels.instance }}
          summary: High invocation total on {{ $labels.instance }}
      - alert: APIHighCPUUsage
        expr: sum (rate (container_cpu_usage_seconds_total{image!="",name=~"^k8s_.*", namespace="openfaas-fn"}[10s])) BY (function_name)
          > 0.4
        for: 5s
        labels:
          service: gateway
          severity: major
          value: '{{$value}}'
        annotations:
          description: High CPU usage on {{ $labels.instance }}
          summary: High CPU usage on {{ $labels.instance }}
      - alert: APIHighMemoryUsage
        expr: sum (container_memory_working_set_bytes{image!="",name=~"^k8s_.*", namespace="openfaas-fn"}) BY (function_name)
          > 30000000
        for: 5s
        labels:
          service: gateway
          severity: major
          value: '{{$value}}'
        annotations:
          description: High Memory usage on {{ $labels.instance }}
          summary: High Memory usage on {{ $labels.instance }}

---

The part of job_name: "cadvisor" is responsible for obtaining the cadvisor metrics of every Kubernetes node and the part of metric_relabel_configs adds function_name label to the pod named find-prime-79c698cd7b-4f67x (a running OpenFass function pod). Then the alerting config expressions focus on the function_name label, which would enable OpenFaas scaling if there is a firing alert.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    proposalquestion

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @rayhero@alexellis@kaikai-liu

        Issue actions
          Auto scaling not only by request/seconds but also considering CPU/Memory usage · Issue #379 · openfaas/faas