Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Multiple-CPUs(or Threads) to improve concurrency. #2188

Closed
10 tasks done
winlinvip opened this issue Feb 5, 2021 · 11 comments
Closed
10 tasks done

Support Multiple-CPUs(or Threads) to improve concurrency. #2188

winlinvip opened this issue Feb 5, 2021 · 11 comments
Assignees
Labels
Discussion Discussion or questions. Enhancement Improvement or enhancement. TransByAI Translated by AI/GPT.
Milestone

Comments

@winlinvip
Copy link
Member

winlinvip commented Feb 5, 2021

Remark

For now, let's put the multi-threading preparation on hold. Although ST already supports multi-threading and the RTC multi-threading part has been almost completed, there are several factors that make me think we should reconsider whether multi-threading is necessary at this stage.

First of all, the multi-threading branch has been deleted from the SRS repository, but it is still preserved in my repository feature/threads, which mainly includes the following commits:

The main reasons for reconsidering multi-threading support are:

  • RTC cascading can extend SRS's capabilities, making it simple and reliable to expand with the Edge/Origin mechanism for live streaming.
  • Single-process concurrency is currently between 1000 and 1200 concurrent connections. After further optimizing QoS, it is estimated that it can maintain around 1000 concurrent connections, which is sufficient for open-source use.
  • Only ST and RTC have more comprehensive multi-threading support, while live streaming and API support are not yet perfect. It requires more effort and is not suitable for our time-limited community to do.
  • The main goal of multi-threading is to expand the single-machine concurrency to the tens of thousands level, which is generally used in large-scale commercial systems. Generally, open-source SFUs do not have this scenario (besides, SRS can use cascading for expansion).

However, simplifying ST and improving its performance can still be considered for merging, including:

Summary

SRS's support for multi-threading is a significant architectural upgrade, essentially aimed at addressing performance issues.

Regarding performance issues, the following points can be expanded:

  • In the live streaming playback scenario, a single-process single-thread can run up to 3000 concurrent connections or higher, as live streaming has no encryption and can directly distribute audio and video data in a one-to-many scenario. Moreover, it can be horizontally scaled through Edge.
  • In the live streaming push scenario, it is impossible to achieve complete horizontal scaling, and the source station cluster scale cannot be very large, as referenced in Cluster: Origin Cluster for Fault Tolarence and Load Balance. #464. If it is a high-bitrate stream, a single-process single-thread can hardly achieve more than 1000 connections.
  • In the RTC scenario, in addition to encryption and QoS, the performance of UDP sending and receiving will also be lower, and it is roughly estimated that it is difficult to reach 500 concurrent connections. This makes SRS no longer a purely IO-intensive server, but an IO and CPU-intensive server.

Why is this issue important?

  • In the RTC scenario, if a single process can only reach 100 or 300 concurrent connections, then 10 times the number of cores is needed. The amount of traffic between these servers is also ten times higher. At this point, the concurrency capability is entirely insufficient.
  • In the live streaming scenario, the current multi-process, Origin, and Edge cluster expansion capabilities can be supported by multi-threading, allowing a single machine to achieve high concurrency, reducing the number of machines to manage, and lowering system complexity.
  • In the monitoring scenario, it is no longer a case of a single stream being consumed by many people, but rather many streams requiring encryption. If a single process does not support enough connections, it will cause many process issues.

Therefore, the multi-threading architecture can be considered a revolution after the multi-coroutine architecture, but this time it is a self-revolution.

Arch

The previous SRS single-threaded architecture (SRS/1/2/3/4):

image

  • Single-process single-threaded architecture, live streaming supports Origin-Edge cluster, and if deployed on a single machine, it is a multi-process single-threaded architecture.
  • RTC is computationally and IO-intensive, so multi-core capabilities are essential, and the main problem is the issue described in this post.
  • Despite this, SRS4 has also made many single-core performance optimizations, as referenced in b431ad7, c5d2027, 14bfc98, and 36ea673

The ultimate goal architecture is horizontally scalable Hybrid threads, also known as low-lock multi-threaded structure (SRS/v5.0.3):

image

  • Merge SRTP thread into Hybrid thread, both threads call OpenSSL simultaneously, posing a risk of Crash.
  • UDP RECV and SEND threads are optional, disabled by default, and Hybrid threads are responsible for sending and receiving data.
  • ST is changed to a thread-local structure (ST#19), with each thread having its isolated ST. It is crucial to ensure that the ST structure of one thread does not pass to another thread, i.e., not reading or writing other threads' FDs.
  • Hybrid threads default to 1, which is almost identical to SRS4's completely single-threaded structure, maintaining structural simplicity. If high performance is needed, multiple threads can be enabled, and the architecture remains essentially unchanged (from one thread to multiple independent threads).
  • Hybrid threads support horizontal scaling, using a multi-port approach. RTC returns different ports through SDP, while RTMP and HTTP require 302 redirection. The specific implementation depends on the documentation.
  • When Hybrid scales horizontally, it still locks connections to a single thread based on the stream, enabling push stream expansion. It can almost scale horizontally to hundreds or thousands of cores, but as the number of cores increases, communication between threads increases, and it is not entirely cost-free, but the cost is relatively small.
  • When Hybrid scales horizontally, in general, multiple push streams and multiple play streams can be well supported, such as 1000 push streams with 10 play streams each, totaling 10,000 streams.

The disadvantages of this architecture:

  • Single-stream playback expansion issue: When Hybrid scales horizontally, since each stream is locked to a single thread, the number of play streams for a single stream is limited by the number of connections supported by a single thread. Solution: In the future, cascading will be used to solve the downstream expansion issue, such as supporting 1000 connections for a single stream on a single machine (regardless of the number of cores), and cascading 1000 servers to support 1 million play streams.
  • Global variable and static variable cleanup issue: Although multi-threading is isolated, they still have a chance to make mistakes through global and static variables. Therefore, all global variables must be checked and modified, either changed to thread-local or thread-safe. This brings risks to stability and is often difficult to troubleshoot when problems arise. Solution: By default, only 1 thread is enabled, allowing for a long enough transition and improvement period.
  • Library thread safety issue: For example, OpenSSL has multi-threading issues, OpenSSL 1.1 claims to be thread-safe, requires modifying the compilation script, and does not use the option -no-threads, but sometimes it may forget to change this option and cause problems. Solution: By default, only 1 SRTP thread is enabled, allowing for a long enough transition and improvement period.

Single-stream playback expansion issue: If you must modify multi-threading to allow a single machine with multiple cores to support playback horizontal scaling, you can have the push stream thread broadcast to the pull stream thread. This change is acceptable, but open-source does not entirely pursue performance. It still needs to maintain a very simple architecture with performance optimization. Personally, I think it is reasonable for a single machine to support 1000 play streams and use cascading to solve expansion, so this optimization will not be used in the future.

Note: Image source is here.

Communication Mechanism

There are two ways for threads to communicate: the first is locked chan, and the second is passing fd. The second can rely on the first.

Both methods should avoid passing audio and video data. Of course, they can be passed, but it is not efficient. For example, you can start a transcoding thread, communicate with chan, and it doesn't require much concurrency.

SRS will have multiple ST threads, which communicate through chan, but they do not pass audio and video data, only some coordination messages.

Currently, SRS's thread communication uses pipe implementation to avoid locks. Therefore, when using it, be aware that it is a low-efficiency mechanism and should not directly pass audio and video packets. It is mainly used for communication between the Master (API) thread and the Hybrid (service) thread, with Hybrid returning SDP to the API.

Thread Types

Each thread will have its ST, and ST is thread-local, i.e., independent and isolated ST.

Remark: The most critical risk and change is to avoid passing FD (or other ST resources) created by one thread to another thread for processing, which will definitely cause problems.

In the end, there will be several types of threads:

  1. Main thread. It mainly manages configuration, manages threads, passes messages, and listens to APIs.
  2. Log thread. Read logs from the queue and write them to disk. To avoid disk IO blocking ST, recording and HLS writing to disk can also be placed in this thread.
  3. One or more Hybrid threads. Listen to network FDs, create epoll, start ST, and serve as the main audio and video business thread. Threads do not communicate with each other. Live streaming uses REUSE PORT, and RTC uses multi-port isolation.
  4. Optional, SRT thread. As it is now, an independent SRT, pushing RTMP to the ST thread via local socket.
  5. Optional, if implementing the SRT protocol yourself, the ST thread can handle SRT clients.
  6. Optional, transcoding or AI thread, pulling audio and video data from the ST thread via local socket, or passing data through Chan, implementing capabilities such as mixing and streaming.

Milestones

4.0 will not enable multi-threading, maintaining single-threaded capabilities.

5.0 will implement most of the multi-threading capabilities, including improving ST's thread-local capabilities. However, Hybrid will only default to 1 thread, and although the process has multiple threads, the overall difference from the previous single-thread is not significant.

6.0 will enable as many threads as there are CPU cores by default, completing the entire multi-threaded architecture transformation.

Differences from Go

Go's multi-threading overhead is too high, and its performance is not sufficient, as it is designed for general services.

With multiple cores, such as 16 cores, Go has about 5 cores for switching. This is because there are locks and data copying between multiple threads, even though chan is used.

In addition, Go is genuinely multi-threaded, requiring constant consideration of competition and thread switching, while SRS is still genuinely single-threaded. Go is more complicated to use, while SRS can still maintain the simplicity of single-threading.

SRS is a multi-threaded and coroutine-based architecture optimized for business, essentially still single-threaded, with threads being essentially unrelated.

Relationship with Source

A single ST thread will have multiple sources.

A source, which is a push stream and its corresponding consumer (playback), is only in one ST thread.

In this way, both push and play are completed in a single ST thread, without the need for locks or switching.

Since the client's URL is unknown when connecting, it is also unknown which stream it belongs to, so it may be accepted by the wrong ST thread, requiring FD migration.

Migrating FD between multiple threads is relatively simple. The difficulty lies in ST, which needs to support multi-threading and consider rebuilding the FD in the new ST thread's epoll when migrating FD. However, this is not particularly difficult, and it is much easier than multi-process.

Why not Multi-process

FD migration between multi-processes is too difficult to implement, and communication between processes is not as easy as communication between threads, nor is it as efficient as threads.

The reason why Nginx uses multi-process is that there is no need for FD migration between multiple processes. So when doing live streaming, NginxRTMP processes push streams to each other, which is too difficult to maintain.

If not migrated, audio and video packets need to be forwarded, and it is definitely better and more suitable for streaming media to migrate FD based on the stream.

Thread Local

Each thread has its own ST, which can be referred to as the Envoy Threading Model, using the C++ thread_local keyword to indicate variables.

I wrote an example SRS: thread-local.cpp, with the following results:

$ ./thread-local
PFN1: tl_g_nn(0x7fbd59504080)=1, g_obj(0x7fbd59504084)=1, gp_obj(0x7fbd59504088,0x7fbd595040a0)=1, gp_obj2(0x7fbd59504090,0x7fbd595040b0)=1
PFN2: tl_g_nn(0x7fbd59604080)=2, g_obj(0x7fbd59604084)=2, gp_obj(0x7fbd59604088,0x7fbd596040a0)=2, gp_obj2(0x7fbd59604090,0x7fbd596040b0)=2
MAIN: tl_g_nn(0x7fbd59704080)=100, g_obj(0x7fbd59704084)=100, gp_obj(0x7fbd59704088,0x7fbd597040a0)=100, gp_obj2(0x7fbd59704090,0x7fbd597040b0)=100

It can be used to modify global variables:

// Global thread local int variable.
thread_local int tl_g_nn = 0;

Including global pointers:

thread_local MyClass g_obj(0);
thread_local MyClass* gp_obj = new MyClass(0);
thread_local MyClass* gp_obj2 = NULL;
MyClass* get_gp_obj2()
{
    if (!gp_obj2) {
        gp_obj2 = new MyClass(0);
    }
    return gp_obj2;
}

The addresses and values of these pointers are different in each thread.

GCC __thread

GCC has extended the keyword __thread, which has the same effect as C++11's thread_local.

A multi-threaded version of ST has been implemented before, using gcc's __thread keyword, referring to toffaletti and ST#19.

UDP Binding

Note: Please note that we ultimately chose to implement RTC multi-thread isolation with multiple ports instead of using UDP binding, so I have collapsed related comments by default.

RTC's UDP is connectionless, and multiple threads can reuse the fd through REUSE_PORT to receive packets sent to the same port.

The kernel will perform a five-tuple binding. When the kernel delivers to a certain listen fd, it will continue to deliver to this fd. Refer to udp-client and udp-server:

Start 3 workers, at 127.0.0.1:8000
listen at 127.0.0.1:8000, fd=3 ok
listen at 127.0.0.1:8000, fd=4 ok
listen at 127.0.0.1:8000, fd=5 ok
fd #5, peer 127.0.0.1:50331, got 13B, Hello world!
fd #5, peer 127.0.0.1:50331, got 13B, Hello world!
fd #5, peer 127.0.0.1:50331, got 13B, Hello world!

Note: There are three fds listening on port 8000 above. After the client 50331 delivers to fd=5, it will continue to deliver to this fd.

UDP Migration

If we receive a client packet from a certain fd, such as 3, and find that this client should be received by another fd, such as 4, we can use connect to bind the delivery relationship.

Refer to the example udp-connect-client.cpp and udp-connect-server.cpp. The server receives the packet and continuously uses other fds to connect. The performance is different on different platforms.

CentOS 7 server, listening on 0.0.0.0:8000, as shown below, can achieve migration twice:

Start 2 workers, at 0.0.0.0:8000
listen at 0.0.0.0:8000, fd=4, migrate_fd=3 ok
listen at 0.0.0.0:8000, fd=3, migrate_fd=4 ok

fd #3, peer 172.16.239.217:37846, got 13B, Hello world!
Transfer 172.16.239.217:37846 from #3 to #4, r0=0, errno=0

fd #4, peer 172.16.239.217:37846, got 13B, Hello world!
Transfer 172.16.239.217:37846 from #4 to #3, r0=0, errno=0

fd #3, peer 172.16.239.217:37846, got 13B, Hello world!
Transfer 172.16.239.217:37846 from #3 to #4, r0=0, errno=0
fd #3, peer 172.16.239.217:37846, got 13B, Hello world!

CentOS 7 server, if bound to a fixed address, such as eth0 or lo, will not migrate:

Start 2 workers, at 172.16.123.121:8000
listen at 172.16.123.121:8000, fd=4, migrate_fd=3 ok
listen at 172.16.123.121:8000, fd=3, migrate_fd=4 ok

fd #3, peer 120.227.88.168:43015, got 13B, Hello world!
Transfer 120.227.88.168:43015 from #3 to #4, r0=0, errno=0

fd #3, peer 120.227.88.168:43015, got 13B, Hello world!
Transfer 120.227.88.168:43015 from #3 to #4, r0=0, errno=0
fd #3, peer 120.227.88.168:43015, got 13B, Hello world!

Note: Linux multiple migrations will not return an error, but will not take effect.

Mac server, regardless of which address is bound, will migrate once:

Start 2 workers, at 127.0.0.1:8000
listen at 127.0.0.1:8000, fd=4, migrate_fd=3 ok
listen at 127.0.0.1:8000, fd=3, migrate_fd=4 ok

fd #3, peer 127.0.0.1:61448, got 13B, Hello world!
Transfer 127.0.0.1:61448 from #3 to #4, r0=0, errno=0

fd #4, peer 127.0.0.1:61448, got 13B, Hello world!
Transfer 127.0.0.1:61448 from #4 to #3, r0=-1, errno=48
fd #4, peer 127.0.0.1:61448, got 13B, Hello world!

Note: On Mac, multiple migrations will return an error [ERRNO 48] ADDRESS ALREADY IN USE.

After connecting with @wasphin, we don't want to migrate. Instead, we hope to bind the 5-tuple to this fd after connecting, so as to avoid other FDs receiving packets.

In this case, a more suitable thread model is:

  1. The UDP port is listened and packets are sent and received by default by the public thread, and the processing thread sends and receives packets through the public thread.
  2. If the processing thread finds that this 5-tuple is not what it should handle, it will transfer the processing to other threads through the message queue.
  3. If the processing thread finds that the packets of this 5-tuple are to be processed by itself, it will open the FD and connect to this address, so that only this thread will directly send and receive packets in the future.

This model is actually a hybrid model:

  1. Most of the time, there is no need to pass packets through locks and inter-thread communication queues.
  2. In a few cases, especially when a new address has just started, packets can be passed through the message queue.
  3. In the early stage of architecture evolution, it is possible not to connect, which means that packets will be passed between threads.

This hybrid model does not rely on UDP connect, but the performance will be very high when Connect works.

In addition, the encryption and decryption problem can also be solved by a similar hybrid model:

  1. Start multiple independent encryption and decryption threads and pass packets through the queue.
  2. If the performance of the working thread is sufficient, it can directly encrypt and decrypt by itself.
  3. In the early stage of architecture evolution, there can be independent encryption and decryption threads, which means that packets will be passed between threads.

What's special is the disk IO thread, which will definitely use the queue to send messages:

  1. The log writing thread collects logs from other threads through the queue and writes them to the disk.
  2. The TS, FLV, and MP4 file writing threads, also known as recording threads, collect the file content to be written through the queue and write the content to the disk.

In the early days, we will still pass packets between multiple threads and divide different threads according to the business. As the evolution progresses, we will gradually eliminate the communication and dependencies between threads and turn them into independent threads that do not rely on each other, achieving higher performance.

@winlinvip winlinvip added the Discussion Discussion or questions. label Feb 5, 2021
@winlinvip
Copy link
Member Author

winlinvip commented Feb 5, 2021

See #2188 (comment)

@winlinvip
Copy link
Member Author

winlinvip commented Feb 5, 2021

See #2188 (comment)

@wasphin

This comment has been minimized.

@winlinvip
Copy link
Member Author

winlinvip commented Feb 23, 2021

See #2188 (comment)

@winlinvip winlinvip changed the title Support Multiple-CPUs to improve cocurrency. Support Multiple-CPUs or Multiple-Threads to improve cocurrency. Feb 23, 2021
@winlinvip winlinvip changed the title Support Multiple-CPUs or Multiple-Threads to improve cocurrency. Support Multiple-CPUs(or Threads) to improve cocurrency. Feb 23, 2021
winlinvip added a commit that referenced this issue Feb 26, 2021
winlinvip added a commit that referenced this issue Mar 2, 2021
winlinvip added a commit that referenced this issue Mar 11, 2021
1. Use SrsThreadPool to execute the hybrid server.
2. Right now, directly run in primordial thread, that is no threads.
3. Define the main APIs of SrsThreadPool.
winlinvip added a commit that referenced this issue Mar 12, 2021
1. Create thread when execute by thread pool.
2. The primordial thread check all threads status.
3. Have not complete the cleanup and stop.
winlinvip added a commit that referenced this issue Mar 12, 2021
1. Create thread when execute by thread pool.
2. The primordial thread check all threads status.
3. Have not complete the cleanup and stop.
winlinvip added a commit that referenced this issue Mar 12, 2021
1. Create thread when execute by thread pool.
2. The primordial thread check all threads status.
3. Have not complete the cleanup and stop.
winlinvip added a commit that referenced this issue Mar 12, 2021
1. Set lock attribute type to PTHREAD_MUTEX_ERRORCHECK.
2. If dead lock, the pthread_mutex_lock return EDEADLK.
3. We assert fail if lock failed.
winlinvip added a commit that referenced this issue Mar 12, 2021
1. Use SrsThreadPool to execute the hybrid server.
2. Right now, directly run in primordial thread, that is no threads.
3. Define the main APIs of SrsThreadPool.
winlinvip added a commit that referenced this issue Mar 12, 2021
1. Create thread when execute by thread pool.
2. The primordial thread check all threads status.
3. Have not complete the cleanup and stop.
winlinvip added a commit that referenced this issue Mar 12, 2021
1. Set lock attribute type to PTHREAD_MUTEX_ERRORCHECK.
2. If dead lock, the pthread_mutex_lock return EDEADLK.
3. We assert fail if lock failed.
winlinvip added a commit that referenced this issue Mar 12, 2021
1. Set lock attribute type to PTHREAD_MUTEX_ERRORCHECK.
2. If dead lock, the pthread_mutex_lock return EDEADLK.
3. We assert fail if lock failed.
winlinvip added a commit that referenced this issue Mar 12, 2021
1. Remove support for reload log configs.
2. Add config for thread pool cycle interval.
3. Add config for log flush interval.
4. Create a SrsAsyncLogManager to create writers.
5. Create a SrsAsyncFileWriter to write file async.
winlinvip added a commit that referenced this issue Mar 12, 2021
1. Wrap std::vector to thread-safe queue.
2. Keep API compatible with std::vector.
3. SrsAsyncFileWriter use thread-safe queue instead.
winlinvip added a commit that referenced this issue Mar 14, 2021
1. Create dual queue, the coroutine queue and thread queue.
2. The coroutine queue cache logs does not require lock.
3. When need to flush, flush the logs from coroutine-queue to thread-queue.
4. Finally, flush thread-queue to disk.
winlinvip added a commit that referenced this issue Mar 14, 2021
1. App/User controls the interval to flush coroutine-queue.
2. Use srs_update_system_time to get time for log.
3. Stat the thread sync in us, in SrsThreadPool.
4. Change default interval for thread to 5s.
winlinvip added a commit that referenced this issue Mar 14, 2021
1. Dual queue for async logs, exists risk.
2. Flush the logs is too slow, because it depends on logs and interval.
winlinvip added a commit that referenced this issue Mar 14, 2021
1. It exists delay for multiple threads.
2. There is overlay for cache of coroutine queues.
3. Risk when other threads write logs.
winlinvip added a commit that referenced this issue Mar 14, 2021
1. If configed the async srtp, use a new object.
2. Allow sync and async srtp, by config.
@winlinvip winlinvip changed the title Support Multiple-CPUs(or Threads) to improve cocurrency. Support Multiple-CPUs(or Threads) to improve concurrency. Mar 14, 2021
winlinvip added a commit that referenced this issue Apr 28, 2021
commit f4872e5
Author: winlin <winlin@vip.126.com>
Date:   Fri Feb 26 09:13:21 2021 +0800

    ST: For #2188: Remove sendmmsg from ST.

commit aaeb891
Author: winlin <winlin@vip.126.com>
Date:   Thu Mar 11 08:09:54 2021 +0800

    ST: Refine utest script.

commit d1ac9da
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 3 11:02:25 2021 +0800

    ST: Support fast utest and coverage

commit 8400115
Author: winlin <winlin@vip.126.com>
Date:   Fri Feb 26 07:02:19 2021 +0800

    ST: Always use unserialized accept for linux or darwin

commit c3686f2
Author: winlin <winlin@vip.126.com>
Date:   Fri Feb 26 06:54:05 2021 +0800

    ST: Refine ARFLAGS by disable the verbose log

commit aaa5c4f
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 08:58:46 2021 +0800

    ST: Stack always grows from top to down.

commit dddd466
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 08:51:31 2021 +0800

    ST: Ignore process fork, for single process only

commit 7906cb5
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 08:50:59 2021 +0800

    ST: Fix build warnings

commit d94921b
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 07:27:45 2021 +0800

    ST: Remove select and poll support, only epoll and kqueue

commit 76d2025
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 07:10:47 2021 +0800

    ST: Remove multiple OS support, except Linux and Darwin.

commit 13c4ba3
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 06:59:35 2021 +0800

    ST: Remove __ia64__ CPU support

commit 46c06e4
Author: winlin <winlin@vip.126.com>
Date:   Wed Feb 24 11:37:27 2021 +0800

    ST: Remove unused files for ST
winlinvip added a commit that referenced this issue Apr 28, 2021
commit
    Remove SEND/RECV/SRTP threads.
    Enable generate_streams by default.
    Support disable circuit breaker.

commit 6c83e89
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 26 09:38:19 2021 +0800

    Threads: Refine circuit breaker

commit 614a781
Author: winlin <winlin@vip.126.com>
Date:   Sun Apr 25 19:52:13 2021 +0800

    Threads-Hybrid: Fix rebase bugs

commit 9c845c5
Author: winlin <winlin@vip.126.com>
Date:   Wed Apr 7 15:53:14 2021 +0800

    Threads-Hybrid: Support multiple hybrid threads, 5.0.3

    1. Support multiple hybrid threads.
    2. Update benchmark data for 1/4/8/32 CPUs.
    3. Update benchmark for Janus.

commit 10edbb5
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 11:45:59 2021 +0800

    Threads-Hybrid: Always response channel with error information

commit 5074a4d
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 11:21:45 2021 +0800

    Threads-Hybrid: Use ST wait and IO for RECV/SEND thread.

commit c8f2ff0
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 10:21:05 2021 +0800

    Threads-Hybrid: Merge api to master thread.

commit 50f03ad
Author: winlin <winlin@vip.126.com>
Date:   Wed Apr 7 13:26:04 2021 +0800

    Threads-Hybrid: Config auto generate stream config for hybrids.

commit 11aada1
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 20:00:16 2021 +0800

    Threads-Hybrid: Process api request one by one

commit 963a0fa
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 19:46:53 2021 +0800

    Threads-Hybrid: Change pps stat to thread-local.

commit 9eb9b19
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 15:21:48 2021 +0800

    Threads-Hybrid: Move acquire pid file from hybrid to pool.

commit 97f6684
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 12:53:17 2021 +0800

    Threads-Hybrid: Schedule connection to the sample hybrid by url

commit eab4f89
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 12:20:38 2021 +0800

    Threads-Hybrid: Change global variables to thread-local or with lock

    1. Thread-Safe: Config subscribes, subscribe or unsubscribe.
    2. Global-Shared: Async SRTP/RECV/SEND/Log use thread-safe objects.
    3. Global-Shared: SRTP and DTLS certificate, without critical data.
    4. Thread-Local: Blackhole, ResourceManager, StreamManager, ObjectCache, by design.
    5. Global-Shared: Log and context, which use thread-safe objects.
    6. Thread-Local: Pithy print for each thread.

commit 7dbac15
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 09:10:16 2021 +0800

    Threads-Hybrid: Add TODO as be thread-local

commit babe8e6
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 08:22:48 2021 +0800

    Threads-Hybrid: Start multiple hybrid threads

commit 3367956
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 07:59:37 2021 +0800

    Threads-Hybrid: Support mulitple hybrid/stream servers

    1. For hybrid RTMP/HTTP/RTC servers.
    2. Config hybrids in threads.
    3. Get hybrid server config with index.

commit 6e2de90
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 21:48:25 2021 +0800

    Threads-Hybrid: Support communicate between threads by chan and slot

    1. Hybrid thread is responder, API thread is initiator.
    2. Responder read message from initiator-slot, write message to responder-slot.
    3. Initiator write message to initiator-slot, read message from responder-slot.
    4. Responder start a coroutine to consume requests and response it.

commit 281350d
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 18:53:32 2021 +0800

    Threads-Hybrid: Extract pipe and pair to communicate between threads

    1. Pipe can be open by any threads, because it's only os FDs.
    2. If pipe is open read/write, it's associated by ST, so we MUST free it by the same thread.
    3. If open pipe in one thread, it's ok to free it directly, without close pipe.
    4. If open read in a thread, then open write in another thread, user MUST close it correctly.

commit 40e59ef
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 16:16:23 2021 +0800

    Threads-Hybrid: Enable RTC play API with bugs

commit 9f4fbdb
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 12:39:28 2021 +0800

    Threads-Hybrid: Init ST for each threads

    1. ST is thread-local now.
    2. MUST init st for each threads.
    3. Do it as early as possible.

commit 2fa3aca
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 10:22:13 2021 +0800

    Threads-Hybrid: Extract API server and threads.

commit dd7c7ad
Author: winlin <winlin@vip.126.com>
Date:   Sun Apr 4 20:44:07 2021 +0800

    Threads-Hybrid: Refine conf file

commit 246d7f5
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 11:55:10 2021 +0800

    Threads-Hybrid: Research for extern and __thread

commit 1a0456a
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 22:44:56 2021 +0800

    Threads: TODO: Sort the packets, to avoid NACK

commit b11f958
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 31 18:37:39 2021 +0800

    Threads: Support multiple threads with locks, #2188. 5.0.2

    1. Threads-Log: Use thread to write and reopen logs.
    2. Threads-SRTP: Support decrypt RTP by async SRTP.
    3. Threads-RECV: Support dedicate thread to recv UDP packets.
    4. Threads: Support cpu affinity for threads.
    5. Threads-RECV: Drop received packet if exceed max queue size.
    6. Threads: Use coroutine to consume recv/srtp packets.
    7. Threads: Support Circuit-Breaker to work in storms.
    8. Threads-SRTP: Support async decrypt RTCP
    9. Threads-SEND: Support async send UDP packets
    10. Threads-SRTP: Use async encrypt SRTP packet
    11. Threads-SEND/RECV: Bind handler to listener to support multiple ports.
    12. Threads-RECV: Support tunnel for recv-srtp.
    13. Threads-SEND: Support tunnel for srtp-send.
    14. Threads: Support circuit-breaker dying threshold

commit 8cf7ead
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 20:54:36 2021 +0800

    Threads: Enable threads support for openssl.

commit 28d8f1a
Author: winlin <winlin@vip.126.com>
Date:   Wed Apr 7 10:29:18 2021 +0800

    Threads: Support multiple SRTP/SEND/RECV threads.

    1. Move received and cooked packets queue to thread entry, that is, each thread has its own queue.
    2. In RECV thread, push received packet to queue of source thread, in thread listener.
    3. In SRTP thread, push cooked packet to queue of source thread, in asyn SRTP task.
    4. In hybrid thread, directly and only consume the packets of self thread.
    5. Sync between SRTP task by lock.

commit a505b6b
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 16:17:08 2021 +0800

    Threads: Directly use hybrid thread to consume messages.

commit 14bc7bc
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 30 16:38:30 2021 +0800

    Threads: Keep alive when got RTP plaintext

commit e15737f
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 29 17:18:04 2021 +0800

    Threads: Support circuit-breaker dying threshold

commit d6a92cb
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 30 14:14:26 2021 +0800

    Threads-SEND: Support tunnel for srtp-send.

commit d282ccd
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 29 12:02:41 2021 +0800

    Threads-RECV: Support tunnel for recv-srtp.

commit eb7ce7f
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 22 19:09:04 2021 +0800

    Threads-RECV: Reset the cache buffer when copy

commit 1235b33
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 22 18:45:12 2021 +0800

    Threads-SEND/RECV: Bind handler to listener to support multiple ports.

commit 615da26
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 18:52:11 2021 +0800

    Threads-SRTP: Use async encrypt SRTP packet

    1. Async SRTP support protect RTCP.
    2. Send packet ignore when encrypt size is 0.
    3. Callback to send packet if encrypt done.

commit a26b926
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 18:43:08 2021 +0800

    Threads-SRTP: Use async encrypt SRTP packet

    1. Async SRTP support protect RTP.
    2. Send packet ignore when encrypt size is 0.
    3. Callback to send packet if encrypt done.

commit 56ffc28
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 18:03:40 2021 +0800

    Threads-SEND: Support async send UDP packets

    1. Support async send UDP by SrsAsyncSendManager.
    2. Copy UDP packet by SrsAsyncUdpPacket.
    3. Support SrsUdpMuxSocket raw sendto.
    4. Config the async send by async_send.

commit b0800f5
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 16:16:09 2021 +0800

    Threads-SRTP: Support async decrypt RTCP

    1. SrsAsyncSRTP support unprotect_rtcp packet.
    2. Extract on_rtcp_plaintext from on_rtcp.

commit 949c80e
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 14:58:34 2021 +0800

    Threads-RECV: Change UDP recv max size from 6k to 1500 bytes.

commit 198dca5
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 13:48:36 2021 +0800

    Threads: Merge recv and srtp consume to one timer.

commit 57b771a
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 11:30:46 2021 +0800

    Threads: Refine variables and do dispose

    1. Rename packets to srtp or received packets.
    2. Add task dispose API, cleanup in future.
    3. If got packets before init AsyncSRTP, return error.
    4. Never free the SRTPTask, dispose it instead.

commit 9a05d24
Author: winlin <winlin@vip.126.com>
Date:   Thu Mar 18 17:33:51 2021 +0800

    Threads: Support Circuit-Breaker to work in storms.

    1. Config the recv queue, drop packet if exceed.
    2. Config the high and critical threshold and pulse of water level.
    3. If critical water level, disable NACK and TWCC.
    4. If high water level, ignore for NACK insert and send.
    5. Support read the CPU of thread.
    6. Refine SrsPps to support r1s sample.

commit 1c90497
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 17 13:53:55 2021 +0800

    Threads: Use coroutine to consume recv/srtp packets.

commit 957034e
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 17 08:08:17 2021 +0800

    Threads: Use thread-local buffer for log

    1. Call SrsThreadPool::setup() in main(),or  each thread starting.
    2. Initialize the thread-local object in SrsThreadPool::setup().
    3. Change shared log buffer to thread-local.

commit 185359f
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 20:29:35 2021 +0800

    Threads-RECV: Show the dropped packets pps.

commit 6fca411
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 20:26:41 2021 +0800

    Threads-RECV: Drop received packet if exceed max queue size.

    1. Print the number of recv/srtp queue packets.
    2. Drop packet if exceed max recv queue size.

commit 9e554ca
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 19:26:44 2021 +0800

    Threads: Support cpu affinity for threads.

    1. Config cpu_affinity in threads.
    2. Default to not set the cpu affinity.
    3. Support set by cpu range 0-63.

commit fa4c9f9
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 11:42:32 2021 +0800

    Threads: Set the threads name display in top.

commit 1c6e941
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 08:30:21 2021 +0800

    Threads-RECV: Refine the stat for SNMP UDP recv/error

    1. Remove the delta of _srs_snmp_udp_stat.
    2. Use _srs_pps_rloss for receive loss rate.
    3. Use _srs_pps_sloss for send loss rate.

commit 85ea39d
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 15 22:51:58 2021 +0800

    Threads-RECV: Support dedicate thread to recv UDP packets.

    1. Use SrsUdpMuxSocket::raw_recvfrom to read, without ST.
    2. Start a UDP recv thread, to recv packets.
    3. Consume UDP packets in RTC server timer.

commit e3686a4
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 15 21:15:28 2021 +0800

    Threads: Fix bug for SRTP and Log thread nanosleep.

commit 88165ba
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 15 20:36:20 2021 +0800

    Threads-SRTP: Support decrypt RTP by async SRTP.

    1. Create dedicate thread for async srtp.
    2. SrsSecurityTransport use async srtp if config on.
    3. Cook SRTP packets in async srtp.
    4. Consume cooked SRTP packets in RTC server timer.

commit f068540
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 21:39:36 2021 +0800

    Threads-SRTP: Config and add files for the async-srtp

    1. If configed the async srtp, use a new object.
    2. Allow sync and async srtp, by config.

commit 2367483
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 22:11:02 2021 +0800

    Threads-Log: Refine stat for sync wait, in log thread.

commit 6a1d6a0
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 18:43:54 2021 +0800

    Threads-Log: Remove dual queue for sys logs.

    1. It exists delay for multiple threads.
    2. There is overlay for cache of coroutine queues.
    3. Risk when other threads write logs.

commit 6ddbc5f
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 18:30:23 2021 +0800

    Threads-Log: Refine comments for global variable.

    1. Dual queue for async logs, exists risk.
    2. Flush the logs is too slow, because it depends on logs and interval.

commit 6718c38
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 10:42:09 2021 +0800

    Threads-Log: Refine dual queue for log thread.

    1. App/User controls the interval to flush coroutine-queue.
    2. Use srs_update_system_time to get time for log.
    3. Stat the thread sync in us, in SrsThreadPool.
    4. Change default interval for thread to 5s.

commit 37aee44
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 08:55:17 2021 +0800

    Threads-Log: Support dual queue cache for async logs.

    1. Create dual queue, the coroutine queue and thread queue.
    2. The coroutine queue cache logs does not require lock.
    3. When need to flush, flush the logs from coroutine-queue to thread-queue.
    4. Finally, flush thread-queue to disk.

commit 4918160
Author: winlin <winlin@vip.126.com>
Date:   Sat Mar 13 07:09:56 2021 +0800

    Threads-Log: Support thread-safe queue SrsThreadQueue.

    1. Wrap std::vector to thread-safe queue.
    2. Keep API compatible with std::vector.
    3. SrsAsyncFileWriter use thread-safe queue instead.

commit 3810f5f
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 18:59:14 2021 +0800

    Threads-Log: Remove utest for reload log configs.

commit bf95fdf
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 18:48:03 2021 +0800

    Threads-Log: Use thread to write and reopen logs.

    1. Remove support for reload log configs.
    2. Add config for thread pool cycle interval.
    3. Add config for log flush interval.
    4. Create a SrsAsyncLogManager to create writers.
    5. Create a SrsAsyncFileWriter to write file async.

commit 0cea382
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 12:02:36 2021 +0800

    Threads-Log: Refine thread lock type to ERRORCHECK.

    1. Set lock attribute type to PTHREAD_MUTEX_ERRORCHECK.
    2. If dead lock, the pthread_mutex_lock return EDEADLK.
    3. We assert fail if lock failed.

commit 668c9c1
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 08:45:10 2021 +0800

    Threads-Log: Run hybrid server in thread.

    1. Create thread when execute by thread pool.
    2. The primordial thread check all threads status.
    3. Have not complete the cleanup and stop.

commit 831b77b
Author: winlin <winlin@vip.126.com>
Date:   Thu Mar 11 18:06:16 2021 +0800

    Threads-Log: Refine API and main workflow.

    1. Use SrsThreadPool to execute the hybrid server.
    2. Right now, directly run in primordial thread, that is no threads.
    3. Define the main APIs of SrsThreadPool.
winlinvip added a commit that referenced this issue Apr 30, 2021
commit f4872e5
Author: winlin <winlin@vip.126.com>
Date:   Fri Feb 26 09:13:21 2021 +0800

    ST: For #2188: Remove sendmmsg from ST.

commit aaeb891
Author: winlin <winlin@vip.126.com>
Date:   Thu Mar 11 08:09:54 2021 +0800

    ST: Refine utest script.

commit d1ac9da
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 3 11:02:25 2021 +0800

    ST: Support fast utest and coverage

commit 8400115
Author: winlin <winlin@vip.126.com>
Date:   Fri Feb 26 07:02:19 2021 +0800

    ST: Always use unserialized accept for linux or darwin

commit c3686f2
Author: winlin <winlin@vip.126.com>
Date:   Fri Feb 26 06:54:05 2021 +0800

    ST: Refine ARFLAGS by disable the verbose log

commit aaa5c4f
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 08:58:46 2021 +0800

    ST: Stack always grows from top to down.

commit dddd466
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 08:51:31 2021 +0800

    ST: Ignore process fork, for single process only

commit 7906cb5
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 08:50:59 2021 +0800

    ST: Fix build warnings

commit d94921b
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 07:27:45 2021 +0800

    ST: Remove select and poll support, only epoll and kqueue

commit 76d2025
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 07:10:47 2021 +0800

    ST: Remove multiple OS support, except Linux and Darwin.

commit 13c4ba3
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 06:59:35 2021 +0800

    ST: Remove __ia64__ CPU support

commit 46c06e4
Author: winlin <winlin@vip.126.com>
Date:   Wed Feb 24 11:37:27 2021 +0800

    ST: Remove unused files for ST
winlinvip added a commit that referenced this issue Apr 30, 2021
commit
    Remove SEND/RECV/SRTP threads.
    Enable generate_streams by default.
    Support disable circuit breaker.

commit 6c83e89
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 26 09:38:19 2021 +0800

    Threads: Refine circuit breaker

commit 614a781
Author: winlin <winlin@vip.126.com>
Date:   Sun Apr 25 19:52:13 2021 +0800

    Threads-Hybrid: Fix rebase bugs

commit 9c845c5
Author: winlin <winlin@vip.126.com>
Date:   Wed Apr 7 15:53:14 2021 +0800

    Threads-Hybrid: Support multiple hybrid threads, 5.0.3

    1. Support multiple hybrid threads.
    2. Update benchmark data for 1/4/8/32 CPUs.
    3. Update benchmark for Janus.

commit 10edbb5
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 11:45:59 2021 +0800

    Threads-Hybrid: Always response channel with error information

commit 5074a4d
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 11:21:45 2021 +0800

    Threads-Hybrid: Use ST wait and IO for RECV/SEND thread.

commit c8f2ff0
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 10:21:05 2021 +0800

    Threads-Hybrid: Merge api to master thread.

commit 50f03ad
Author: winlin <winlin@vip.126.com>
Date:   Wed Apr 7 13:26:04 2021 +0800

    Threads-Hybrid: Config auto generate stream config for hybrids.

commit 11aada1
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 20:00:16 2021 +0800

    Threads-Hybrid: Process api request one by one

commit 963a0fa
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 19:46:53 2021 +0800

    Threads-Hybrid: Change pps stat to thread-local.

commit 9eb9b19
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 15:21:48 2021 +0800

    Threads-Hybrid: Move acquire pid file from hybrid to pool.

commit 97f6684
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 12:53:17 2021 +0800

    Threads-Hybrid: Schedule connection to the sample hybrid by url

commit eab4f89
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 12:20:38 2021 +0800

    Threads-Hybrid: Change global variables to thread-local or with lock

    1. Thread-Safe: Config subscribes, subscribe or unsubscribe.
    2. Global-Shared: Async SRTP/RECV/SEND/Log use thread-safe objects.
    3. Global-Shared: SRTP and DTLS certificate, without critical data.
    4. Thread-Local: Blackhole, ResourceManager, StreamManager, ObjectCache, by design.
    5. Global-Shared: Log and context, which use thread-safe objects.
    6. Thread-Local: Pithy print for each thread.

commit 7dbac15
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 09:10:16 2021 +0800

    Threads-Hybrid: Add TODO as be thread-local

commit babe8e6
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 08:22:48 2021 +0800

    Threads-Hybrid: Start multiple hybrid threads

commit 3367956
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 07:59:37 2021 +0800

    Threads-Hybrid: Support mulitple hybrid/stream servers

    1. For hybrid RTMP/HTTP/RTC servers.
    2. Config hybrids in threads.
    3. Get hybrid server config with index.

commit 6e2de90
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 21:48:25 2021 +0800

    Threads-Hybrid: Support communicate between threads by chan and slot

    1. Hybrid thread is responder, API thread is initiator.
    2. Responder read message from initiator-slot, write message to responder-slot.
    3. Initiator write message to initiator-slot, read message from responder-slot.
    4. Responder start a coroutine to consume requests and response it.

commit 281350d
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 18:53:32 2021 +0800

    Threads-Hybrid: Extract pipe and pair to communicate between threads

    1. Pipe can be open by any threads, because it's only os FDs.
    2. If pipe is open read/write, it's associated by ST, so we MUST free it by the same thread.
    3. If open pipe in one thread, it's ok to free it directly, without close pipe.
    4. If open read in a thread, then open write in another thread, user MUST close it correctly.

commit 40e59ef
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 16:16:23 2021 +0800

    Threads-Hybrid: Enable RTC play API with bugs

commit 9f4fbdb
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 12:39:28 2021 +0800

    Threads-Hybrid: Init ST for each threads

    1. ST is thread-local now.
    2. MUST init st for each threads.
    3. Do it as early as possible.

commit 2fa3aca
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 10:22:13 2021 +0800

    Threads-Hybrid: Extract API server and threads.

commit dd7c7ad
Author: winlin <winlin@vip.126.com>
Date:   Sun Apr 4 20:44:07 2021 +0800

    Threads-Hybrid: Refine conf file

commit 246d7f5
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 11:55:10 2021 +0800

    Threads-Hybrid: Research for extern and __thread

commit 1a0456a
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 22:44:56 2021 +0800

    Threads: TODO: Sort the packets, to avoid NACK

commit b11f958
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 31 18:37:39 2021 +0800

    Threads: Support multiple threads with locks, #2188. 5.0.2

    1. Threads-Log: Use thread to write and reopen logs.
    2. Threads-SRTP: Support decrypt RTP by async SRTP.
    3. Threads-RECV: Support dedicate thread to recv UDP packets.
    4. Threads: Support cpu affinity for threads.
    5. Threads-RECV: Drop received packet if exceed max queue size.
    6. Threads: Use coroutine to consume recv/srtp packets.
    7. Threads: Support Circuit-Breaker to work in storms.
    8. Threads-SRTP: Support async decrypt RTCP
    9. Threads-SEND: Support async send UDP packets
    10. Threads-SRTP: Use async encrypt SRTP packet
    11. Threads-SEND/RECV: Bind handler to listener to support multiple ports.
    12. Threads-RECV: Support tunnel for recv-srtp.
    13. Threads-SEND: Support tunnel for srtp-send.
    14. Threads: Support circuit-breaker dying threshold

commit 8cf7ead
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 20:54:36 2021 +0800

    Threads: Enable threads support for openssl.

commit 28d8f1a
Author: winlin <winlin@vip.126.com>
Date:   Wed Apr 7 10:29:18 2021 +0800

    Threads: Support multiple SRTP/SEND/RECV threads.

    1. Move received and cooked packets queue to thread entry, that is, each thread has its own queue.
    2. In RECV thread, push received packet to queue of source thread, in thread listener.
    3. In SRTP thread, push cooked packet to queue of source thread, in asyn SRTP task.
    4. In hybrid thread, directly and only consume the packets of self thread.
    5. Sync between SRTP task by lock.

commit a505b6b
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 16:17:08 2021 +0800

    Threads: Directly use hybrid thread to consume messages.

commit 14bc7bc
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 30 16:38:30 2021 +0800

    Threads: Keep alive when got RTP plaintext

commit e15737f
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 29 17:18:04 2021 +0800

    Threads: Support circuit-breaker dying threshold

commit d6a92cb
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 30 14:14:26 2021 +0800

    Threads-SEND: Support tunnel for srtp-send.

commit d282ccd
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 29 12:02:41 2021 +0800

    Threads-RECV: Support tunnel for recv-srtp.

commit eb7ce7f
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 22 19:09:04 2021 +0800

    Threads-RECV: Reset the cache buffer when copy

commit 1235b33
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 22 18:45:12 2021 +0800

    Threads-SEND/RECV: Bind handler to listener to support multiple ports.

commit 615da26
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 18:52:11 2021 +0800

    Threads-SRTP: Use async encrypt SRTP packet

    1. Async SRTP support protect RTCP.
    2. Send packet ignore when encrypt size is 0.
    3. Callback to send packet if encrypt done.

commit a26b926
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 18:43:08 2021 +0800

    Threads-SRTP: Use async encrypt SRTP packet

    1. Async SRTP support protect RTP.
    2. Send packet ignore when encrypt size is 0.
    3. Callback to send packet if encrypt done.

commit 56ffc28
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 18:03:40 2021 +0800

    Threads-SEND: Support async send UDP packets

    1. Support async send UDP by SrsAsyncSendManager.
    2. Copy UDP packet by SrsAsyncUdpPacket.
    3. Support SrsUdpMuxSocket raw sendto.
    4. Config the async send by async_send.

commit b0800f5
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 16:16:09 2021 +0800

    Threads-SRTP: Support async decrypt RTCP

    1. SrsAsyncSRTP support unprotect_rtcp packet.
    2. Extract on_rtcp_plaintext from on_rtcp.

commit 949c80e
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 14:58:34 2021 +0800

    Threads-RECV: Change UDP recv max size from 6k to 1500 bytes.

commit 198dca5
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 13:48:36 2021 +0800

    Threads: Merge recv and srtp consume to one timer.

commit 57b771a
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 11:30:46 2021 +0800

    Threads: Refine variables and do dispose

    1. Rename packets to srtp or received packets.
    2. Add task dispose API, cleanup in future.
    3. If got packets before init AsyncSRTP, return error.
    4. Never free the SRTPTask, dispose it instead.

commit 9a05d24
Author: winlin <winlin@vip.126.com>
Date:   Thu Mar 18 17:33:51 2021 +0800

    Threads: Support Circuit-Breaker to work in storms.

    1. Config the recv queue, drop packet if exceed.
    2. Config the high and critical threshold and pulse of water level.
    3. If critical water level, disable NACK and TWCC.
    4. If high water level, ignore for NACK insert and send.
    5. Support read the CPU of thread.
    6. Refine SrsPps to support r1s sample.

commit 1c90497
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 17 13:53:55 2021 +0800

    Threads: Use coroutine to consume recv/srtp packets.

commit 957034e
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 17 08:08:17 2021 +0800

    Threads: Use thread-local buffer for log

    1. Call SrsThreadPool::setup() in main(),or  each thread starting.
    2. Initialize the thread-local object in SrsThreadPool::setup().
    3. Change shared log buffer to thread-local.

commit 185359f
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 20:29:35 2021 +0800

    Threads-RECV: Show the dropped packets pps.

commit 6fca411
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 20:26:41 2021 +0800

    Threads-RECV: Drop received packet if exceed max queue size.

    1. Print the number of recv/srtp queue packets.
    2. Drop packet if exceed max recv queue size.

commit 9e554ca
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 19:26:44 2021 +0800

    Threads: Support cpu affinity for threads.

    1. Config cpu_affinity in threads.
    2. Default to not set the cpu affinity.
    3. Support set by cpu range 0-63.

commit fa4c9f9
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 11:42:32 2021 +0800

    Threads: Set the threads name display in top.

commit 1c6e941
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 08:30:21 2021 +0800

    Threads-RECV: Refine the stat for SNMP UDP recv/error

    1. Remove the delta of _srs_snmp_udp_stat.
    2. Use _srs_pps_rloss for receive loss rate.
    3. Use _srs_pps_sloss for send loss rate.

commit 85ea39d
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 15 22:51:58 2021 +0800

    Threads-RECV: Support dedicate thread to recv UDP packets.

    1. Use SrsUdpMuxSocket::raw_recvfrom to read, without ST.
    2. Start a UDP recv thread, to recv packets.
    3. Consume UDP packets in RTC server timer.

commit e3686a4
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 15 21:15:28 2021 +0800

    Threads: Fix bug for SRTP and Log thread nanosleep.

commit 88165ba
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 15 20:36:20 2021 +0800

    Threads-SRTP: Support decrypt RTP by async SRTP.

    1. Create dedicate thread for async srtp.
    2. SrsSecurityTransport use async srtp if config on.
    3. Cook SRTP packets in async srtp.
    4. Consume cooked SRTP packets in RTC server timer.

commit f068540
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 21:39:36 2021 +0800

    Threads-SRTP: Config and add files for the async-srtp

    1. If configed the async srtp, use a new object.
    2. Allow sync and async srtp, by config.

commit 2367483
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 22:11:02 2021 +0800

    Threads-Log: Refine stat for sync wait, in log thread.

commit 6a1d6a0
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 18:43:54 2021 +0800

    Threads-Log: Remove dual queue for sys logs.

    1. It exists delay for multiple threads.
    2. There is overlay for cache of coroutine queues.
    3. Risk when other threads write logs.

commit 6ddbc5f
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 18:30:23 2021 +0800

    Threads-Log: Refine comments for global variable.

    1. Dual queue for async logs, exists risk.
    2. Flush the logs is too slow, because it depends on logs and interval.

commit 6718c38
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 10:42:09 2021 +0800

    Threads-Log: Refine dual queue for log thread.

    1. App/User controls the interval to flush coroutine-queue.
    2. Use srs_update_system_time to get time for log.
    3. Stat the thread sync in us, in SrsThreadPool.
    4. Change default interval for thread to 5s.

commit 37aee44
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 08:55:17 2021 +0800

    Threads-Log: Support dual queue cache for async logs.

    1. Create dual queue, the coroutine queue and thread queue.
    2. The coroutine queue cache logs does not require lock.
    3. When need to flush, flush the logs from coroutine-queue to thread-queue.
    4. Finally, flush thread-queue to disk.

commit 4918160
Author: winlin <winlin@vip.126.com>
Date:   Sat Mar 13 07:09:56 2021 +0800

    Threads-Log: Support thread-safe queue SrsThreadQueue.

    1. Wrap std::vector to thread-safe queue.
    2. Keep API compatible with std::vector.
    3. SrsAsyncFileWriter use thread-safe queue instead.

commit 3810f5f
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 18:59:14 2021 +0800

    Threads-Log: Remove utest for reload log configs.

commit bf95fdf
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 18:48:03 2021 +0800

    Threads-Log: Use thread to write and reopen logs.

    1. Remove support for reload log configs.
    2. Add config for thread pool cycle interval.
    3. Add config for log flush interval.
    4. Create a SrsAsyncLogManager to create writers.
    5. Create a SrsAsyncFileWriter to write file async.

commit 0cea382
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 12:02:36 2021 +0800

    Threads-Log: Refine thread lock type to ERRORCHECK.

    1. Set lock attribute type to PTHREAD_MUTEX_ERRORCHECK.
    2. If dead lock, the pthread_mutex_lock return EDEADLK.
    3. We assert fail if lock failed.

commit 668c9c1
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 08:45:10 2021 +0800

    Threads-Log: Run hybrid server in thread.

    1. Create thread when execute by thread pool.
    2. The primordial thread check all threads status.
    3. Have not complete the cleanup and stop.

commit 831b77b
Author: winlin <winlin@vip.126.com>
Date:   Thu Mar 11 18:06:16 2021 +0800

    Threads-Log: Refine API and main workflow.

    1. Use SrsThreadPool to execute the hybrid server.
    2. Right now, directly run in primordial thread, that is no threads.
    3. Define the main APIs of SrsThreadPool.
winlinvip added a commit that referenced this issue May 1, 2021
commit f4872e5
Author: winlin <winlin@vip.126.com>
Date:   Fri Feb 26 09:13:21 2021 +0800

    ST: For #2188: Remove sendmmsg from ST.

commit aaeb891
Author: winlin <winlin@vip.126.com>
Date:   Thu Mar 11 08:09:54 2021 +0800

    ST: Refine utest script.

commit d1ac9da
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 3 11:02:25 2021 +0800

    ST: Support fast utest and coverage

commit 8400115
Author: winlin <winlin@vip.126.com>
Date:   Fri Feb 26 07:02:19 2021 +0800

    ST: Always use unserialized accept for linux or darwin

commit c3686f2
Author: winlin <winlin@vip.126.com>
Date:   Fri Feb 26 06:54:05 2021 +0800

    ST: Refine ARFLAGS by disable the verbose log

commit aaa5c4f
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 08:58:46 2021 +0800

    ST: Stack always grows from top to down.

commit dddd466
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 08:51:31 2021 +0800

    ST: Ignore process fork, for single process only

commit 7906cb5
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 08:50:59 2021 +0800

    ST: Fix build warnings

commit d94921b
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 07:27:45 2021 +0800

    ST: Remove select and poll support, only epoll and kqueue

commit 76d2025
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 07:10:47 2021 +0800

    ST: Remove multiple OS support, except Linux and Darwin.

commit 13c4ba3
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 06:59:35 2021 +0800

    ST: Remove __ia64__ CPU support

commit 46c06e4
Author: winlin <winlin@vip.126.com>
Date:   Wed Feb 24 11:37:27 2021 +0800

    ST: Remove unused files for ST
winlinvip added a commit that referenced this issue May 1, 2021
commit
    Remove SEND/RECV/SRTP threads.
    Enable generate_streams by default.
    Support disable circuit breaker.

commit 6c83e89
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 26 09:38:19 2021 +0800

    Threads: Refine circuit breaker

commit 614a781
Author: winlin <winlin@vip.126.com>
Date:   Sun Apr 25 19:52:13 2021 +0800

    Threads-Hybrid: Fix rebase bugs

commit 9c845c5
Author: winlin <winlin@vip.126.com>
Date:   Wed Apr 7 15:53:14 2021 +0800

    Threads-Hybrid: Support multiple hybrid threads, 5.0.3

    1. Support multiple hybrid threads.
    2. Update benchmark data for 1/4/8/32 CPUs.
    3. Update benchmark for Janus.

commit 10edbb5
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 11:45:59 2021 +0800

    Threads-Hybrid: Always response channel with error information

commit 5074a4d
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 11:21:45 2021 +0800

    Threads-Hybrid: Use ST wait and IO for RECV/SEND thread.

commit c8f2ff0
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 10:21:05 2021 +0800

    Threads-Hybrid: Merge api to master thread.

commit 50f03ad
Author: winlin <winlin@vip.126.com>
Date:   Wed Apr 7 13:26:04 2021 +0800

    Threads-Hybrid: Config auto generate stream config for hybrids.

commit 11aada1
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 20:00:16 2021 +0800

    Threads-Hybrid: Process api request one by one

commit 963a0fa
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 19:46:53 2021 +0800

    Threads-Hybrid: Change pps stat to thread-local.

commit 9eb9b19
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 15:21:48 2021 +0800

    Threads-Hybrid: Move acquire pid file from hybrid to pool.

commit 97f6684
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 12:53:17 2021 +0800

    Threads-Hybrid: Schedule connection to the sample hybrid by url

commit eab4f89
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 12:20:38 2021 +0800

    Threads-Hybrid: Change global variables to thread-local or with lock

    1. Thread-Safe: Config subscribes, subscribe or unsubscribe.
    2. Global-Shared: Async SRTP/RECV/SEND/Log use thread-safe objects.
    3. Global-Shared: SRTP and DTLS certificate, without critical data.
    4. Thread-Local: Blackhole, ResourceManager, StreamManager, ObjectCache, by design.
    5. Global-Shared: Log and context, which use thread-safe objects.
    6. Thread-Local: Pithy print for each thread.

commit 7dbac15
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 09:10:16 2021 +0800

    Threads-Hybrid: Add TODO as be thread-local

commit babe8e6
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 08:22:48 2021 +0800

    Threads-Hybrid: Start multiple hybrid threads

commit 3367956
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 07:59:37 2021 +0800

    Threads-Hybrid: Support mulitple hybrid/stream servers

    1. For hybrid RTMP/HTTP/RTC servers.
    2. Config hybrids in threads.
    3. Get hybrid server config with index.

commit 6e2de90
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 21:48:25 2021 +0800

    Threads-Hybrid: Support communicate between threads by chan and slot

    1. Hybrid thread is responder, API thread is initiator.
    2. Responder read message from initiator-slot, write message to responder-slot.
    3. Initiator write message to initiator-slot, read message from responder-slot.
    4. Responder start a coroutine to consume requests and response it.

commit 281350d
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 18:53:32 2021 +0800

    Threads-Hybrid: Extract pipe and pair to communicate between threads

    1. Pipe can be open by any threads, because it's only os FDs.
    2. If pipe is open read/write, it's associated by ST, so we MUST free it by the same thread.
    3. If open pipe in one thread, it's ok to free it directly, without close pipe.
    4. If open read in a thread, then open write in another thread, user MUST close it correctly.

commit 40e59ef
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 16:16:23 2021 +0800

    Threads-Hybrid: Enable RTC play API with bugs

commit 9f4fbdb
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 12:39:28 2021 +0800

    Threads-Hybrid: Init ST for each threads

    1. ST is thread-local now.
    2. MUST init st for each threads.
    3. Do it as early as possible.

commit 2fa3aca
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 10:22:13 2021 +0800

    Threads-Hybrid: Extract API server and threads.

commit dd7c7ad
Author: winlin <winlin@vip.126.com>
Date:   Sun Apr 4 20:44:07 2021 +0800

    Threads-Hybrid: Refine conf file

commit 246d7f5
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 11:55:10 2021 +0800

    Threads-Hybrid: Research for extern and __thread

commit 1a0456a
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 22:44:56 2021 +0800

    Threads: TODO: Sort the packets, to avoid NACK

commit b11f958
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 31 18:37:39 2021 +0800

    Threads: Support multiple threads with locks, #2188. 5.0.2

    1. Threads-Log: Use thread to write and reopen logs.
    2. Threads-SRTP: Support decrypt RTP by async SRTP.
    3. Threads-RECV: Support dedicate thread to recv UDP packets.
    4. Threads: Support cpu affinity for threads.
    5. Threads-RECV: Drop received packet if exceed max queue size.
    6. Threads: Use coroutine to consume recv/srtp packets.
    7. Threads: Support Circuit-Breaker to work in storms.
    8. Threads-SRTP: Support async decrypt RTCP
    9. Threads-SEND: Support async send UDP packets
    10. Threads-SRTP: Use async encrypt SRTP packet
    11. Threads-SEND/RECV: Bind handler to listener to support multiple ports.
    12. Threads-RECV: Support tunnel for recv-srtp.
    13. Threads-SEND: Support tunnel for srtp-send.
    14. Threads: Support circuit-breaker dying threshold

commit 8cf7ead
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 20:54:36 2021 +0800

    Threads: Enable threads support for openssl.

commit 28d8f1a
Author: winlin <winlin@vip.126.com>
Date:   Wed Apr 7 10:29:18 2021 +0800

    Threads: Support multiple SRTP/SEND/RECV threads.

    1. Move received and cooked packets queue to thread entry, that is, each thread has its own queue.
    2. In RECV thread, push received packet to queue of source thread, in thread listener.
    3. In SRTP thread, push cooked packet to queue of source thread, in asyn SRTP task.
    4. In hybrid thread, directly and only consume the packets of self thread.
    5. Sync between SRTP task by lock.

commit a505b6b
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 16:17:08 2021 +0800

    Threads: Directly use hybrid thread to consume messages.

commit 14bc7bc
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 30 16:38:30 2021 +0800

    Threads: Keep alive when got RTP plaintext

commit e15737f
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 29 17:18:04 2021 +0800

    Threads: Support circuit-breaker dying threshold

commit d6a92cb
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 30 14:14:26 2021 +0800

    Threads-SEND: Support tunnel for srtp-send.

commit d282ccd
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 29 12:02:41 2021 +0800

    Threads-RECV: Support tunnel for recv-srtp.

commit eb7ce7f
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 22 19:09:04 2021 +0800

    Threads-RECV: Reset the cache buffer when copy

commit 1235b33
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 22 18:45:12 2021 +0800

    Threads-SEND/RECV: Bind handler to listener to support multiple ports.

commit 615da26
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 18:52:11 2021 +0800

    Threads-SRTP: Use async encrypt SRTP packet

    1. Async SRTP support protect RTCP.
    2. Send packet ignore when encrypt size is 0.
    3. Callback to send packet if encrypt done.

commit a26b926
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 18:43:08 2021 +0800

    Threads-SRTP: Use async encrypt SRTP packet

    1. Async SRTP support protect RTP.
    2. Send packet ignore when encrypt size is 0.
    3. Callback to send packet if encrypt done.

commit 56ffc28
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 18:03:40 2021 +0800

    Threads-SEND: Support async send UDP packets

    1. Support async send UDP by SrsAsyncSendManager.
    2. Copy UDP packet by SrsAsyncUdpPacket.
    3. Support SrsUdpMuxSocket raw sendto.
    4. Config the async send by async_send.

commit b0800f5
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 16:16:09 2021 +0800

    Threads-SRTP: Support async decrypt RTCP

    1. SrsAsyncSRTP support unprotect_rtcp packet.
    2. Extract on_rtcp_plaintext from on_rtcp.

commit 949c80e
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 14:58:34 2021 +0800

    Threads-RECV: Change UDP recv max size from 6k to 1500 bytes.

commit 198dca5
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 13:48:36 2021 +0800

    Threads: Merge recv and srtp consume to one timer.

commit 57b771a
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 11:30:46 2021 +0800

    Threads: Refine variables and do dispose

    1. Rename packets to srtp or received packets.
    2. Add task dispose API, cleanup in future.
    3. If got packets before init AsyncSRTP, return error.
    4. Never free the SRTPTask, dispose it instead.

commit 9a05d24
Author: winlin <winlin@vip.126.com>
Date:   Thu Mar 18 17:33:51 2021 +0800

    Threads: Support Circuit-Breaker to work in storms.

    1. Config the recv queue, drop packet if exceed.
    2. Config the high and critical threshold and pulse of water level.
    3. If critical water level, disable NACK and TWCC.
    4. If high water level, ignore for NACK insert and send.
    5. Support read the CPU of thread.
    6. Refine SrsPps to support r1s sample.

commit 1c90497
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 17 13:53:55 2021 +0800

    Threads: Use coroutine to consume recv/srtp packets.

commit 957034e
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 17 08:08:17 2021 +0800

    Threads: Use thread-local buffer for log

    1. Call SrsThreadPool::setup() in main(),or  each thread starting.
    2. Initialize the thread-local object in SrsThreadPool::setup().
    3. Change shared log buffer to thread-local.

commit 185359f
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 20:29:35 2021 +0800

    Threads-RECV: Show the dropped packets pps.

commit 6fca411
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 20:26:41 2021 +0800

    Threads-RECV: Drop received packet if exceed max queue size.

    1. Print the number of recv/srtp queue packets.
    2. Drop packet if exceed max recv queue size.

commit 9e554ca
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 19:26:44 2021 +0800

    Threads: Support cpu affinity for threads.

    1. Config cpu_affinity in threads.
    2. Default to not set the cpu affinity.
    3. Support set by cpu range 0-63.

commit fa4c9f9
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 11:42:32 2021 +0800

    Threads: Set the threads name display in top.

commit 1c6e941
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 08:30:21 2021 +0800

    Threads-RECV: Refine the stat for SNMP UDP recv/error

    1. Remove the delta of _srs_snmp_udp_stat.
    2. Use _srs_pps_rloss for receive loss rate.
    3. Use _srs_pps_sloss for send loss rate.

commit 85ea39d
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 15 22:51:58 2021 +0800

    Threads-RECV: Support dedicate thread to recv UDP packets.

    1. Use SrsUdpMuxSocket::raw_recvfrom to read, without ST.
    2. Start a UDP recv thread, to recv packets.
    3. Consume UDP packets in RTC server timer.

commit e3686a4
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 15 21:15:28 2021 +0800

    Threads: Fix bug for SRTP and Log thread nanosleep.

commit 88165ba
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 15 20:36:20 2021 +0800

    Threads-SRTP: Support decrypt RTP by async SRTP.

    1. Create dedicate thread for async srtp.
    2. SrsSecurityTransport use async srtp if config on.
    3. Cook SRTP packets in async srtp.
    4. Consume cooked SRTP packets in RTC server timer.

commit f068540
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 21:39:36 2021 +0800

    Threads-SRTP: Config and add files for the async-srtp

    1. If configed the async srtp, use a new object.
    2. Allow sync and async srtp, by config.

commit 2367483
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 22:11:02 2021 +0800

    Threads-Log: Refine stat for sync wait, in log thread.

commit 6a1d6a0
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 18:43:54 2021 +0800

    Threads-Log: Remove dual queue for sys logs.

    1. It exists delay for multiple threads.
    2. There is overlay for cache of coroutine queues.
    3. Risk when other threads write logs.

commit 6ddbc5f
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 18:30:23 2021 +0800

    Threads-Log: Refine comments for global variable.

    1. Dual queue for async logs, exists risk.
    2. Flush the logs is too slow, because it depends on logs and interval.

commit 6718c38
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 10:42:09 2021 +0800

    Threads-Log: Refine dual queue for log thread.

    1. App/User controls the interval to flush coroutine-queue.
    2. Use srs_update_system_time to get time for log.
    3. Stat the thread sync in us, in SrsThreadPool.
    4. Change default interval for thread to 5s.

commit 37aee44
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 08:55:17 2021 +0800

    Threads-Log: Support dual queue cache for async logs.

    1. Create dual queue, the coroutine queue and thread queue.
    2. The coroutine queue cache logs does not require lock.
    3. When need to flush, flush the logs from coroutine-queue to thread-queue.
    4. Finally, flush thread-queue to disk.

commit 4918160
Author: winlin <winlin@vip.126.com>
Date:   Sat Mar 13 07:09:56 2021 +0800

    Threads-Log: Support thread-safe queue SrsThreadQueue.

    1. Wrap std::vector to thread-safe queue.
    2. Keep API compatible with std::vector.
    3. SrsAsyncFileWriter use thread-safe queue instead.

commit 3810f5f
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 18:59:14 2021 +0800

    Threads-Log: Remove utest for reload log configs.

commit bf95fdf
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 18:48:03 2021 +0800

    Threads-Log: Use thread to write and reopen logs.

    1. Remove support for reload log configs.
    2. Add config for thread pool cycle interval.
    3. Add config for log flush interval.
    4. Create a SrsAsyncLogManager to create writers.
    5. Create a SrsAsyncFileWriter to write file async.

commit 0cea382
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 12:02:36 2021 +0800

    Threads-Log: Refine thread lock type to ERRORCHECK.

    1. Set lock attribute type to PTHREAD_MUTEX_ERRORCHECK.
    2. If dead lock, the pthread_mutex_lock return EDEADLK.
    3. We assert fail if lock failed.

commit 668c9c1
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 08:45:10 2021 +0800

    Threads-Log: Run hybrid server in thread.

    1. Create thread when execute by thread pool.
    2. The primordial thread check all threads status.
    3. Have not complete the cleanup and stop.

commit 831b77b
Author: winlin <winlin@vip.126.com>
Date:   Thu Mar 11 18:06:16 2021 +0800

    Threads-Log: Refine API and main workflow.

    1. Use SrsThreadPool to execute the hybrid server.
    2. Right now, directly run in primordial thread, that is no threads.
    3. Define the main APIs of SrsThreadPool.
winlinvip added a commit that referenced this issue May 1, 2021
commit f4872e5
Author: winlin <winlin@vip.126.com>
Date:   Fri Feb 26 09:13:21 2021 +0800

    ST: For #2188: Remove sendmmsg from ST.

commit aaeb891
Author: winlin <winlin@vip.126.com>
Date:   Thu Mar 11 08:09:54 2021 +0800

    ST: Refine utest script.

commit d1ac9da
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 3 11:02:25 2021 +0800

    ST: Support fast utest and coverage

commit 8400115
Author: winlin <winlin@vip.126.com>
Date:   Fri Feb 26 07:02:19 2021 +0800

    ST: Always use unserialized accept for linux or darwin

commit c3686f2
Author: winlin <winlin@vip.126.com>
Date:   Fri Feb 26 06:54:05 2021 +0800

    ST: Refine ARFLAGS by disable the verbose log

commit aaa5c4f
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 08:58:46 2021 +0800

    ST: Stack always grows from top to down.

commit dddd466
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 08:51:31 2021 +0800

    ST: Ignore process fork, for single process only

commit 7906cb5
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 08:50:59 2021 +0800

    ST: Fix build warnings

commit d94921b
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 07:27:45 2021 +0800

    ST: Remove select and poll support, only epoll and kqueue

commit 76d2025
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 07:10:47 2021 +0800

    ST: Remove multiple OS support, except Linux and Darwin.

commit 13c4ba3
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 06:59:35 2021 +0800

    ST: Remove __ia64__ CPU support

commit 46c06e4
Author: winlin <winlin@vip.126.com>
Date:   Wed Feb 24 11:37:27 2021 +0800

    ST: Remove unused files for ST
winlinvip added a commit that referenced this issue May 1, 2021
commit
    Remove SEND/RECV/SRTP threads.
    Enable generate_streams by default.
    Support disable circuit breaker.

commit 6c83e89
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 26 09:38:19 2021 +0800

    Threads: Refine circuit breaker

commit 614a781
Author: winlin <winlin@vip.126.com>
Date:   Sun Apr 25 19:52:13 2021 +0800

    Threads-Hybrid: Fix rebase bugs

commit 9c845c5
Author: winlin <winlin@vip.126.com>
Date:   Wed Apr 7 15:53:14 2021 +0800

    Threads-Hybrid: Support multiple hybrid threads, 5.0.3

    1. Support multiple hybrid threads.
    2. Update benchmark data for 1/4/8/32 CPUs.
    3. Update benchmark for Janus.

commit 10edbb5
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 11:45:59 2021 +0800

    Threads-Hybrid: Always response channel with error information

commit 5074a4d
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 11:21:45 2021 +0800

    Threads-Hybrid: Use ST wait and IO for RECV/SEND thread.

commit c8f2ff0
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 10:21:05 2021 +0800

    Threads-Hybrid: Merge api to master thread.

commit 50f03ad
Author: winlin <winlin@vip.126.com>
Date:   Wed Apr 7 13:26:04 2021 +0800

    Threads-Hybrid: Config auto generate stream config for hybrids.

commit 11aada1
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 20:00:16 2021 +0800

    Threads-Hybrid: Process api request one by one

commit 963a0fa
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 19:46:53 2021 +0800

    Threads-Hybrid: Change pps stat to thread-local.

commit 9eb9b19
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 15:21:48 2021 +0800

    Threads-Hybrid: Move acquire pid file from hybrid to pool.

commit 97f6684
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 12:53:17 2021 +0800

    Threads-Hybrid: Schedule connection to the sample hybrid by url

commit eab4f89
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 12:20:38 2021 +0800

    Threads-Hybrid: Change global variables to thread-local or with lock

    1. Thread-Safe: Config subscribes, subscribe or unsubscribe.
    2. Global-Shared: Async SRTP/RECV/SEND/Log use thread-safe objects.
    3. Global-Shared: SRTP and DTLS certificate, without critical data.
    4. Thread-Local: Blackhole, ResourceManager, StreamManager, ObjectCache, by design.
    5. Global-Shared: Log and context, which use thread-safe objects.
    6. Thread-Local: Pithy print for each thread.

commit 7dbac15
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 09:10:16 2021 +0800

    Threads-Hybrid: Add TODO as be thread-local

commit babe8e6
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 08:22:48 2021 +0800

    Threads-Hybrid: Start multiple hybrid threads

commit 3367956
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 07:59:37 2021 +0800

    Threads-Hybrid: Support mulitple hybrid/stream servers

    1. For hybrid RTMP/HTTP/RTC servers.
    2. Config hybrids in threads.
    3. Get hybrid server config with index.

commit 6e2de90
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 21:48:25 2021 +0800

    Threads-Hybrid: Support communicate between threads by chan and slot

    1. Hybrid thread is responder, API thread is initiator.
    2. Responder read message from initiator-slot, write message to responder-slot.
    3. Initiator write message to initiator-slot, read message from responder-slot.
    4. Responder start a coroutine to consume requests and response it.

commit 281350d
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 18:53:32 2021 +0800

    Threads-Hybrid: Extract pipe and pair to communicate between threads

    1. Pipe can be open by any threads, because it's only os FDs.
    2. If pipe is open read/write, it's associated by ST, so we MUST free it by the same thread.
    3. If open pipe in one thread, it's ok to free it directly, without close pipe.
    4. If open read in a thread, then open write in another thread, user MUST close it correctly.

commit 40e59ef
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 16:16:23 2021 +0800

    Threads-Hybrid: Enable RTC play API with bugs

commit 9f4fbdb
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 12:39:28 2021 +0800

    Threads-Hybrid: Init ST for each threads

    1. ST is thread-local now.
    2. MUST init st for each threads.
    3. Do it as early as possible.

commit 2fa3aca
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 10:22:13 2021 +0800

    Threads-Hybrid: Extract API server and threads.

commit dd7c7ad
Author: winlin <winlin@vip.126.com>
Date:   Sun Apr 4 20:44:07 2021 +0800

    Threads-Hybrid: Refine conf file

commit 246d7f5
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 11:55:10 2021 +0800

    Threads-Hybrid: Research for extern and __thread

commit 1a0456a
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 22:44:56 2021 +0800

    Threads: TODO: Sort the packets, to avoid NACK

commit b11f958
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 31 18:37:39 2021 +0800

    Threads: Support multiple threads with locks, #2188. 5.0.2

    1. Threads-Log: Use thread to write and reopen logs.
    2. Threads-SRTP: Support decrypt RTP by async SRTP.
    3. Threads-RECV: Support dedicate thread to recv UDP packets.
    4. Threads: Support cpu affinity for threads.
    5. Threads-RECV: Drop received packet if exceed max queue size.
    6. Threads: Use coroutine to consume recv/srtp packets.
    7. Threads: Support Circuit-Breaker to work in storms.
    8. Threads-SRTP: Support async decrypt RTCP
    9. Threads-SEND: Support async send UDP packets
    10. Threads-SRTP: Use async encrypt SRTP packet
    11. Threads-SEND/RECV: Bind handler to listener to support multiple ports.
    12. Threads-RECV: Support tunnel for recv-srtp.
    13. Threads-SEND: Support tunnel for srtp-send.
    14. Threads: Support circuit-breaker dying threshold

commit 8cf7ead
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 20:54:36 2021 +0800

    Threads: Enable threads support for openssl.

commit 28d8f1a
Author: winlin <winlin@vip.126.com>
Date:   Wed Apr 7 10:29:18 2021 +0800

    Threads: Support multiple SRTP/SEND/RECV threads.

    1. Move received and cooked packets queue to thread entry, that is, each thread has its own queue.
    2. In RECV thread, push received packet to queue of source thread, in thread listener.
    3. In SRTP thread, push cooked packet to queue of source thread, in asyn SRTP task.
    4. In hybrid thread, directly and only consume the packets of self thread.
    5. Sync between SRTP task by lock.

commit a505b6b
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 16:17:08 2021 +0800

    Threads: Directly use hybrid thread to consume messages.

commit 14bc7bc
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 30 16:38:30 2021 +0800

    Threads: Keep alive when got RTP plaintext

commit e15737f
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 29 17:18:04 2021 +0800

    Threads: Support circuit-breaker dying threshold

commit d6a92cb
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 30 14:14:26 2021 +0800

    Threads-SEND: Support tunnel for srtp-send.

commit d282ccd
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 29 12:02:41 2021 +0800

    Threads-RECV: Support tunnel for recv-srtp.

commit eb7ce7f
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 22 19:09:04 2021 +0800

    Threads-RECV: Reset the cache buffer when copy

commit 1235b33
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 22 18:45:12 2021 +0800

    Threads-SEND/RECV: Bind handler to listener to support multiple ports.

commit 615da26
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 18:52:11 2021 +0800

    Threads-SRTP: Use async encrypt SRTP packet

    1. Async SRTP support protect RTCP.
    2. Send packet ignore when encrypt size is 0.
    3. Callback to send packet if encrypt done.

commit a26b926
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 18:43:08 2021 +0800

    Threads-SRTP: Use async encrypt SRTP packet

    1. Async SRTP support protect RTP.
    2. Send packet ignore when encrypt size is 0.
    3. Callback to send packet if encrypt done.

commit 56ffc28
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 18:03:40 2021 +0800

    Threads-SEND: Support async send UDP packets

    1. Support async send UDP by SrsAsyncSendManager.
    2. Copy UDP packet by SrsAsyncUdpPacket.
    3. Support SrsUdpMuxSocket raw sendto.
    4. Config the async send by async_send.

commit b0800f5
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 16:16:09 2021 +0800

    Threads-SRTP: Support async decrypt RTCP

    1. SrsAsyncSRTP support unprotect_rtcp packet.
    2. Extract on_rtcp_plaintext from on_rtcp.

commit 949c80e
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 14:58:34 2021 +0800

    Threads-RECV: Change UDP recv max size from 6k to 1500 bytes.

commit 198dca5
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 13:48:36 2021 +0800

    Threads: Merge recv and srtp consume to one timer.

commit 57b771a
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 11:30:46 2021 +0800

    Threads: Refine variables and do dispose

    1. Rename packets to srtp or received packets.
    2. Add task dispose API, cleanup in future.
    3. If got packets before init AsyncSRTP, return error.
    4. Never free the SRTPTask, dispose it instead.

commit 9a05d24
Author: winlin <winlin@vip.126.com>
Date:   Thu Mar 18 17:33:51 2021 +0800

    Threads: Support Circuit-Breaker to work in storms.

    1. Config the recv queue, drop packet if exceed.
    2. Config the high and critical threshold and pulse of water level.
    3. If critical water level, disable NACK and TWCC.
    4. If high water level, ignore for NACK insert and send.
    5. Support read the CPU of thread.
    6. Refine SrsPps to support r1s sample.

commit 1c90497
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 17 13:53:55 2021 +0800

    Threads: Use coroutine to consume recv/srtp packets.

commit 957034e
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 17 08:08:17 2021 +0800

    Threads: Use thread-local buffer for log

    1. Call SrsThreadPool::setup() in main(),or  each thread starting.
    2. Initialize the thread-local object in SrsThreadPool::setup().
    3. Change shared log buffer to thread-local.

commit 185359f
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 20:29:35 2021 +0800

    Threads-RECV: Show the dropped packets pps.

commit 6fca411
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 20:26:41 2021 +0800

    Threads-RECV: Drop received packet if exceed max queue size.

    1. Print the number of recv/srtp queue packets.
    2. Drop packet if exceed max recv queue size.

commit 9e554ca
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 19:26:44 2021 +0800

    Threads: Support cpu affinity for threads.

    1. Config cpu_affinity in threads.
    2. Default to not set the cpu affinity.
    3. Support set by cpu range 0-63.

commit fa4c9f9
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 11:42:32 2021 +0800

    Threads: Set the threads name display in top.

commit 1c6e941
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 08:30:21 2021 +0800

    Threads-RECV: Refine the stat for SNMP UDP recv/error

    1. Remove the delta of _srs_snmp_udp_stat.
    2. Use _srs_pps_rloss for receive loss rate.
    3. Use _srs_pps_sloss for send loss rate.

commit 85ea39d
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 15 22:51:58 2021 +0800

    Threads-RECV: Support dedicate thread to recv UDP packets.

    1. Use SrsUdpMuxSocket::raw_recvfrom to read, without ST.
    2. Start a UDP recv thread, to recv packets.
    3. Consume UDP packets in RTC server timer.

commit e3686a4
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 15 21:15:28 2021 +0800

    Threads: Fix bug for SRTP and Log thread nanosleep.

commit 88165ba
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 15 20:36:20 2021 +0800

    Threads-SRTP: Support decrypt RTP by async SRTP.

    1. Create dedicate thread for async srtp.
    2. SrsSecurityTransport use async srtp if config on.
    3. Cook SRTP packets in async srtp.
    4. Consume cooked SRTP packets in RTC server timer.

commit f068540
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 21:39:36 2021 +0800

    Threads-SRTP: Config and add files for the async-srtp

    1. If configed the async srtp, use a new object.
    2. Allow sync and async srtp, by config.

commit 2367483
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 22:11:02 2021 +0800

    Threads-Log: Refine stat for sync wait, in log thread.

commit 6a1d6a0
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 18:43:54 2021 +0800

    Threads-Log: Remove dual queue for sys logs.

    1. It exists delay for multiple threads.
    2. There is overlay for cache of coroutine queues.
    3. Risk when other threads write logs.

commit 6ddbc5f
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 18:30:23 2021 +0800

    Threads-Log: Refine comments for global variable.

    1. Dual queue for async logs, exists risk.
    2. Flush the logs is too slow, because it depends on logs and interval.

commit 6718c38
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 10:42:09 2021 +0800

    Threads-Log: Refine dual queue for log thread.

    1. App/User controls the interval to flush coroutine-queue.
    2. Use srs_update_system_time to get time for log.
    3. Stat the thread sync in us, in SrsThreadPool.
    4. Change default interval for thread to 5s.

commit 37aee44
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 08:55:17 2021 +0800

    Threads-Log: Support dual queue cache for async logs.

    1. Create dual queue, the coroutine queue and thread queue.
    2. The coroutine queue cache logs does not require lock.
    3. When need to flush, flush the logs from coroutine-queue to thread-queue.
    4. Finally, flush thread-queue to disk.

commit 4918160
Author: winlin <winlin@vip.126.com>
Date:   Sat Mar 13 07:09:56 2021 +0800

    Threads-Log: Support thread-safe queue SrsThreadQueue.

    1. Wrap std::vector to thread-safe queue.
    2. Keep API compatible with std::vector.
    3. SrsAsyncFileWriter use thread-safe queue instead.

commit 3810f5f
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 18:59:14 2021 +0800

    Threads-Log: Remove utest for reload log configs.

commit bf95fdf
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 18:48:03 2021 +0800

    Threads-Log: Use thread to write and reopen logs.

    1. Remove support for reload log configs.
    2. Add config for thread pool cycle interval.
    3. Add config for log flush interval.
    4. Create a SrsAsyncLogManager to create writers.
    5. Create a SrsAsyncFileWriter to write file async.

commit 0cea382
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 12:02:36 2021 +0800

    Threads-Log: Refine thread lock type to ERRORCHECK.

    1. Set lock attribute type to PTHREAD_MUTEX_ERRORCHECK.
    2. If dead lock, the pthread_mutex_lock return EDEADLK.
    3. We assert fail if lock failed.

commit 668c9c1
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 08:45:10 2021 +0800

    Threads-Log: Run hybrid server in thread.

    1. Create thread when execute by thread pool.
    2. The primordial thread check all threads status.
    3. Have not complete the cleanup and stop.

commit 831b77b
Author: winlin <winlin@vip.126.com>
Date:   Thu Mar 11 18:06:16 2021 +0800

    Threads-Log: Refine API and main workflow.

    1. Use SrsThreadPool to execute the hybrid server.
    2. Right now, directly run in primordial thread, that is no threads.
    3. Define the main APIs of SrsThreadPool.
winlinvip added a commit that referenced this issue May 6, 2021
commit f4872e5
Author: winlin <winlin@vip.126.com>
Date:   Fri Feb 26 09:13:21 2021 +0800

    ST: For #2188: Remove sendmmsg from ST.

commit aaeb891
Author: winlin <winlin@vip.126.com>
Date:   Thu Mar 11 08:09:54 2021 +0800

    ST: Refine utest script.

commit d1ac9da
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 3 11:02:25 2021 +0800

    ST: Support fast utest and coverage

commit 8400115
Author: winlin <winlin@vip.126.com>
Date:   Fri Feb 26 07:02:19 2021 +0800

    ST: Always use unserialized accept for linux or darwin

commit c3686f2
Author: winlin <winlin@vip.126.com>
Date:   Fri Feb 26 06:54:05 2021 +0800

    ST: Refine ARFLAGS by disable the verbose log

commit aaa5c4f
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 08:58:46 2021 +0800

    ST: Stack always grows from top to down.

commit dddd466
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 08:51:31 2021 +0800

    ST: Ignore process fork, for single process only

commit 7906cb5
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 08:50:59 2021 +0800

    ST: Fix build warnings

commit d94921b
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 07:27:45 2021 +0800

    ST: Remove select and poll support, only epoll and kqueue

commit 76d2025
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 07:10:47 2021 +0800

    ST: Remove multiple OS support, except Linux and Darwin.

commit 13c4ba3
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 06:59:35 2021 +0800

    ST: Remove __ia64__ CPU support

commit 46c06e4
Author: winlin <winlin@vip.126.com>
Date:   Wed Feb 24 11:37:27 2021 +0800

    ST: Remove unused files for ST
winlinvip added a commit that referenced this issue May 6, 2021
commit
    Remove SEND/RECV/SRTP threads.
    Enable generate_streams by default.
    Support disable circuit breaker.

commit 6c83e89
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 26 09:38:19 2021 +0800

    Threads: Refine circuit breaker

commit 614a781
Author: winlin <winlin@vip.126.com>
Date:   Sun Apr 25 19:52:13 2021 +0800

    Threads-Hybrid: Fix rebase bugs

commit 9c845c5
Author: winlin <winlin@vip.126.com>
Date:   Wed Apr 7 15:53:14 2021 +0800

    Threads-Hybrid: Support multiple hybrid threads, 5.0.3

    1. Support multiple hybrid threads.
    2. Update benchmark data for 1/4/8/32 CPUs.
    3. Update benchmark for Janus.

commit 10edbb5
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 11:45:59 2021 +0800

    Threads-Hybrid: Always response channel with error information

commit 5074a4d
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 11:21:45 2021 +0800

    Threads-Hybrid: Use ST wait and IO for RECV/SEND thread.

commit c8f2ff0
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 10:21:05 2021 +0800

    Threads-Hybrid: Merge api to master thread.

commit 50f03ad
Author: winlin <winlin@vip.126.com>
Date:   Wed Apr 7 13:26:04 2021 +0800

    Threads-Hybrid: Config auto generate stream config for hybrids.

commit 11aada1
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 20:00:16 2021 +0800

    Threads-Hybrid: Process api request one by one

commit 963a0fa
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 19:46:53 2021 +0800

    Threads-Hybrid: Change pps stat to thread-local.

commit 9eb9b19
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 15:21:48 2021 +0800

    Threads-Hybrid: Move acquire pid file from hybrid to pool.

commit 97f6684
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 12:53:17 2021 +0800

    Threads-Hybrid: Schedule connection to the sample hybrid by url

commit eab4f89
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 12:20:38 2021 +0800

    Threads-Hybrid: Change global variables to thread-local or with lock

    1. Thread-Safe: Config subscribes, subscribe or unsubscribe.
    2. Global-Shared: Async SRTP/RECV/SEND/Log use thread-safe objects.
    3. Global-Shared: SRTP and DTLS certificate, without critical data.
    4. Thread-Local: Blackhole, ResourceManager, StreamManager, ObjectCache, by design.
    5. Global-Shared: Log and context, which use thread-safe objects.
    6. Thread-Local: Pithy print for each thread.

commit 7dbac15
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 09:10:16 2021 +0800

    Threads-Hybrid: Add TODO as be thread-local

commit babe8e6
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 08:22:48 2021 +0800

    Threads-Hybrid: Start multiple hybrid threads

commit 3367956
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 07:59:37 2021 +0800

    Threads-Hybrid: Support mulitple hybrid/stream servers

    1. For hybrid RTMP/HTTP/RTC servers.
    2. Config hybrids in threads.
    3. Get hybrid server config with index.

commit 6e2de90
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 21:48:25 2021 +0800

    Threads-Hybrid: Support communicate between threads by chan and slot

    1. Hybrid thread is responder, API thread is initiator.
    2. Responder read message from initiator-slot, write message to responder-slot.
    3. Initiator write message to initiator-slot, read message from responder-slot.
    4. Responder start a coroutine to consume requests and response it.

commit 281350d
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 18:53:32 2021 +0800

    Threads-Hybrid: Extract pipe and pair to communicate between threads

    1. Pipe can be open by any threads, because it's only os FDs.
    2. If pipe is open read/write, it's associated by ST, so we MUST free it by the same thread.
    3. If open pipe in one thread, it's ok to free it directly, without close pipe.
    4. If open read in a thread, then open write in another thread, user MUST close it correctly.

commit 40e59ef
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 16:16:23 2021 +0800

    Threads-Hybrid: Enable RTC play API with bugs

commit 9f4fbdb
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 12:39:28 2021 +0800

    Threads-Hybrid: Init ST for each threads

    1. ST is thread-local now.
    2. MUST init st for each threads.
    3. Do it as early as possible.

commit 2fa3aca
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 10:22:13 2021 +0800

    Threads-Hybrid: Extract API server and threads.

commit dd7c7ad
Author: winlin <winlin@vip.126.com>
Date:   Sun Apr 4 20:44:07 2021 +0800

    Threads-Hybrid: Refine conf file

commit 246d7f5
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 11:55:10 2021 +0800

    Threads-Hybrid: Research for extern and __thread

commit 1a0456a
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 22:44:56 2021 +0800

    Threads: TODO: Sort the packets, to avoid NACK

commit b11f958
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 31 18:37:39 2021 +0800

    Threads: Support multiple threads with locks, #2188. 5.0.2

    1. Threads-Log: Use thread to write and reopen logs.
    2. Threads-SRTP: Support decrypt RTP by async SRTP.
    3. Threads-RECV: Support dedicate thread to recv UDP packets.
    4. Threads: Support cpu affinity for threads.
    5. Threads-RECV: Drop received packet if exceed max queue size.
    6. Threads: Use coroutine to consume recv/srtp packets.
    7. Threads: Support Circuit-Breaker to work in storms.
    8. Threads-SRTP: Support async decrypt RTCP
    9. Threads-SEND: Support async send UDP packets
    10. Threads-SRTP: Use async encrypt SRTP packet
    11. Threads-SEND/RECV: Bind handler to listener to support multiple ports.
    12. Threads-RECV: Support tunnel for recv-srtp.
    13. Threads-SEND: Support tunnel for srtp-send.
    14. Threads: Support circuit-breaker dying threshold

commit 8cf7ead
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 20:54:36 2021 +0800

    Threads: Enable threads support for openssl.

commit 28d8f1a
Author: winlin <winlin@vip.126.com>
Date:   Wed Apr 7 10:29:18 2021 +0800

    Threads: Support multiple SRTP/SEND/RECV threads.

    1. Move received and cooked packets queue to thread entry, that is, each thread has its own queue.
    2. In RECV thread, push received packet to queue of source thread, in thread listener.
    3. In SRTP thread, push cooked packet to queue of source thread, in asyn SRTP task.
    4. In hybrid thread, directly and only consume the packets of self thread.
    5. Sync between SRTP task by lock.

commit a505b6b
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 16:17:08 2021 +0800

    Threads: Directly use hybrid thread to consume messages.

commit 14bc7bc
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 30 16:38:30 2021 +0800

    Threads: Keep alive when got RTP plaintext

commit e15737f
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 29 17:18:04 2021 +0800

    Threads: Support circuit-breaker dying threshold

commit d6a92cb
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 30 14:14:26 2021 +0800

    Threads-SEND: Support tunnel for srtp-send.

commit d282ccd
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 29 12:02:41 2021 +0800

    Threads-RECV: Support tunnel for recv-srtp.

commit eb7ce7f
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 22 19:09:04 2021 +0800

    Threads-RECV: Reset the cache buffer when copy

commit 1235b33
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 22 18:45:12 2021 +0800

    Threads-SEND/RECV: Bind handler to listener to support multiple ports.

commit 615da26
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 18:52:11 2021 +0800

    Threads-SRTP: Use async encrypt SRTP packet

    1. Async SRTP support protect RTCP.
    2. Send packet ignore when encrypt size is 0.
    3. Callback to send packet if encrypt done.

commit a26b926
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 18:43:08 2021 +0800

    Threads-SRTP: Use async encrypt SRTP packet

    1. Async SRTP support protect RTP.
    2. Send packet ignore when encrypt size is 0.
    3. Callback to send packet if encrypt done.

commit 56ffc28
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 18:03:40 2021 +0800

    Threads-SEND: Support async send UDP packets

    1. Support async send UDP by SrsAsyncSendManager.
    2. Copy UDP packet by SrsAsyncUdpPacket.
    3. Support SrsUdpMuxSocket raw sendto.
    4. Config the async send by async_send.

commit b0800f5
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 16:16:09 2021 +0800

    Threads-SRTP: Support async decrypt RTCP

    1. SrsAsyncSRTP support unprotect_rtcp packet.
    2. Extract on_rtcp_plaintext from on_rtcp.

commit 949c80e
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 14:58:34 2021 +0800

    Threads-RECV: Change UDP recv max size from 6k to 1500 bytes.

commit 198dca5
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 13:48:36 2021 +0800

    Threads: Merge recv and srtp consume to one timer.

commit 57b771a
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 11:30:46 2021 +0800

    Threads: Refine variables and do dispose

    1. Rename packets to srtp or received packets.
    2. Add task dispose API, cleanup in future.
    3. If got packets before init AsyncSRTP, return error.
    4. Never free the SRTPTask, dispose it instead.

commit 9a05d24
Author: winlin <winlin@vip.126.com>
Date:   Thu Mar 18 17:33:51 2021 +0800

    Threads: Support Circuit-Breaker to work in storms.

    1. Config the recv queue, drop packet if exceed.
    2. Config the high and critical threshold and pulse of water level.
    3. If critical water level, disable NACK and TWCC.
    4. If high water level, ignore for NACK insert and send.
    5. Support read the CPU of thread.
    6. Refine SrsPps to support r1s sample.

commit 1c90497
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 17 13:53:55 2021 +0800

    Threads: Use coroutine to consume recv/srtp packets.

commit 957034e
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 17 08:08:17 2021 +0800

    Threads: Use thread-local buffer for log

    1. Call SrsThreadPool::setup() in main(),or  each thread starting.
    2. Initialize the thread-local object in SrsThreadPool::setup().
    3. Change shared log buffer to thread-local.

commit 185359f
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 20:29:35 2021 +0800

    Threads-RECV: Show the dropped packets pps.

commit 6fca411
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 20:26:41 2021 +0800

    Threads-RECV: Drop received packet if exceed max queue size.

    1. Print the number of recv/srtp queue packets.
    2. Drop packet if exceed max recv queue size.

commit 9e554ca
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 19:26:44 2021 +0800

    Threads: Support cpu affinity for threads.

    1. Config cpu_affinity in threads.
    2. Default to not set the cpu affinity.
    3. Support set by cpu range 0-63.

commit fa4c9f9
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 11:42:32 2021 +0800

    Threads: Set the threads name display in top.

commit 1c6e941
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 08:30:21 2021 +0800

    Threads-RECV: Refine the stat for SNMP UDP recv/error

    1. Remove the delta of _srs_snmp_udp_stat.
    2. Use _srs_pps_rloss for receive loss rate.
    3. Use _srs_pps_sloss for send loss rate.

commit 85ea39d
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 15 22:51:58 2021 +0800

    Threads-RECV: Support dedicate thread to recv UDP packets.

    1. Use SrsUdpMuxSocket::raw_recvfrom to read, without ST.
    2. Start a UDP recv thread, to recv packets.
    3. Consume UDP packets in RTC server timer.

commit e3686a4
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 15 21:15:28 2021 +0800

    Threads: Fix bug for SRTP and Log thread nanosleep.

commit 88165ba
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 15 20:36:20 2021 +0800

    Threads-SRTP: Support decrypt RTP by async SRTP.

    1. Create dedicate thread for async srtp.
    2. SrsSecurityTransport use async srtp if config on.
    3. Cook SRTP packets in async srtp.
    4. Consume cooked SRTP packets in RTC server timer.

commit f068540
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 21:39:36 2021 +0800

    Threads-SRTP: Config and add files for the async-srtp

    1. If configed the async srtp, use a new object.
    2. Allow sync and async srtp, by config.

commit 2367483
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 22:11:02 2021 +0800

    Threads-Log: Refine stat for sync wait, in log thread.

commit 6a1d6a0
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 18:43:54 2021 +0800

    Threads-Log: Remove dual queue for sys logs.

    1. It exists delay for multiple threads.
    2. There is overlay for cache of coroutine queues.
    3. Risk when other threads write logs.

commit 6ddbc5f
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 18:30:23 2021 +0800

    Threads-Log: Refine comments for global variable.

    1. Dual queue for async logs, exists risk.
    2. Flush the logs is too slow, because it depends on logs and interval.

commit 6718c38
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 10:42:09 2021 +0800

    Threads-Log: Refine dual queue for log thread.

    1. App/User controls the interval to flush coroutine-queue.
    2. Use srs_update_system_time to get time for log.
    3. Stat the thread sync in us, in SrsThreadPool.
    4. Change default interval for thread to 5s.

commit 37aee44
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 08:55:17 2021 +0800

    Threads-Log: Support dual queue cache for async logs.

    1. Create dual queue, the coroutine queue and thread queue.
    2. The coroutine queue cache logs does not require lock.
    3. When need to flush, flush the logs from coroutine-queue to thread-queue.
    4. Finally, flush thread-queue to disk.

commit 4918160
Author: winlin <winlin@vip.126.com>
Date:   Sat Mar 13 07:09:56 2021 +0800

    Threads-Log: Support thread-safe queue SrsThreadQueue.

    1. Wrap std::vector to thread-safe queue.
    2. Keep API compatible with std::vector.
    3. SrsAsyncFileWriter use thread-safe queue instead.

commit 3810f5f
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 18:59:14 2021 +0800

    Threads-Log: Remove utest for reload log configs.

commit bf95fdf
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 18:48:03 2021 +0800

    Threads-Log: Use thread to write and reopen logs.

    1. Remove support for reload log configs.
    2. Add config for thread pool cycle interval.
    3. Add config for log flush interval.
    4. Create a SrsAsyncLogManager to create writers.
    5. Create a SrsAsyncFileWriter to write file async.

commit 0cea382
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 12:02:36 2021 +0800

    Threads-Log: Refine thread lock type to ERRORCHECK.

    1. Set lock attribute type to PTHREAD_MUTEX_ERRORCHECK.
    2. If dead lock, the pthread_mutex_lock return EDEADLK.
    3. We assert fail if lock failed.

commit 668c9c1
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 08:45:10 2021 +0800

    Threads-Log: Run hybrid server in thread.

    1. Create thread when execute by thread pool.
    2. The primordial thread check all threads status.
    3. Have not complete the cleanup and stop.

commit 831b77b
Author: winlin <winlin@vip.126.com>
Date:   Thu Mar 11 18:06:16 2021 +0800

    Threads-Log: Refine API and main workflow.

    1. Use SrsThreadPool to execute the hybrid server.
    2. Right now, directly run in primordial thread, that is no threads.
    3. Define the main APIs of SrsThreadPool.
winlinvip added a commit that referenced this issue May 7, 2021
commit f4872e5
Author: winlin <winlin@vip.126.com>
Date:   Fri Feb 26 09:13:21 2021 +0800

    ST: For #2188: Remove sendmmsg from ST.

commit aaeb891
Author: winlin <winlin@vip.126.com>
Date:   Thu Mar 11 08:09:54 2021 +0800

    ST: Refine utest script.

commit d1ac9da
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 3 11:02:25 2021 +0800

    ST: Support fast utest and coverage

commit 8400115
Author: winlin <winlin@vip.126.com>
Date:   Fri Feb 26 07:02:19 2021 +0800

    ST: Always use unserialized accept for linux or darwin

commit c3686f2
Author: winlin <winlin@vip.126.com>
Date:   Fri Feb 26 06:54:05 2021 +0800

    ST: Refine ARFLAGS by disable the verbose log

commit aaa5c4f
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 08:58:46 2021 +0800

    ST: Stack always grows from top to down.

commit dddd466
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 08:51:31 2021 +0800

    ST: Ignore process fork, for single process only

commit 7906cb5
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 08:50:59 2021 +0800

    ST: Fix build warnings

commit d94921b
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 07:27:45 2021 +0800

    ST: Remove select and poll support, only epoll and kqueue

commit 76d2025
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 07:10:47 2021 +0800

    ST: Remove multiple OS support, except Linux and Darwin.

commit 13c4ba3
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 06:59:35 2021 +0800

    ST: Remove __ia64__ CPU support

commit 46c06e4
Author: winlin <winlin@vip.126.com>
Date:   Wed Feb 24 11:37:27 2021 +0800

    ST: Remove unused files for ST
winlinvip added a commit that referenced this issue May 7, 2021
commit
    Remove SEND/RECV/SRTP threads.
    Enable generate_streams by default.
    Support disable circuit breaker.

commit 6c83e89
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 26 09:38:19 2021 +0800

    Threads: Refine circuit breaker

commit 614a781
Author: winlin <winlin@vip.126.com>
Date:   Sun Apr 25 19:52:13 2021 +0800

    Threads-Hybrid: Fix rebase bugs

commit 9c845c5
Author: winlin <winlin@vip.126.com>
Date:   Wed Apr 7 15:53:14 2021 +0800

    Threads-Hybrid: Support multiple hybrid threads, 5.0.3

    1. Support multiple hybrid threads.
    2. Update benchmark data for 1/4/8/32 CPUs.
    3. Update benchmark for Janus.

commit 10edbb5
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 11:45:59 2021 +0800

    Threads-Hybrid: Always response channel with error information

commit 5074a4d
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 11:21:45 2021 +0800

    Threads-Hybrid: Use ST wait and IO for RECV/SEND thread.

commit c8f2ff0
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 10:21:05 2021 +0800

    Threads-Hybrid: Merge api to master thread.

commit 50f03ad
Author: winlin <winlin@vip.126.com>
Date:   Wed Apr 7 13:26:04 2021 +0800

    Threads-Hybrid: Config auto generate stream config for hybrids.

commit 11aada1
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 20:00:16 2021 +0800

    Threads-Hybrid: Process api request one by one

commit 963a0fa
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 19:46:53 2021 +0800

    Threads-Hybrid: Change pps stat to thread-local.

commit 9eb9b19
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 15:21:48 2021 +0800

    Threads-Hybrid: Move acquire pid file from hybrid to pool.

commit 97f6684
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 12:53:17 2021 +0800

    Threads-Hybrid: Schedule connection to the sample hybrid by url

commit eab4f89
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 12:20:38 2021 +0800

    Threads-Hybrid: Change global variables to thread-local or with lock

    1. Thread-Safe: Config subscribes, subscribe or unsubscribe.
    2. Global-Shared: Async SRTP/RECV/SEND/Log use thread-safe objects.
    3. Global-Shared: SRTP and DTLS certificate, without critical data.
    4. Thread-Local: Blackhole, ResourceManager, StreamManager, ObjectCache, by design.
    5. Global-Shared: Log and context, which use thread-safe objects.
    6. Thread-Local: Pithy print for each thread.

commit 7dbac15
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 09:10:16 2021 +0800

    Threads-Hybrid: Add TODO as be thread-local

commit babe8e6
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 08:22:48 2021 +0800

    Threads-Hybrid: Start multiple hybrid threads

commit 3367956
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 07:59:37 2021 +0800

    Threads-Hybrid: Support mulitple hybrid/stream servers

    1. For hybrid RTMP/HTTP/RTC servers.
    2. Config hybrids in threads.
    3. Get hybrid server config with index.

commit 6e2de90
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 21:48:25 2021 +0800

    Threads-Hybrid: Support communicate between threads by chan and slot

    1. Hybrid thread is responder, API thread is initiator.
    2. Responder read message from initiator-slot, write message to responder-slot.
    3. Initiator write message to initiator-slot, read message from responder-slot.
    4. Responder start a coroutine to consume requests and response it.

commit 281350d
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 18:53:32 2021 +0800

    Threads-Hybrid: Extract pipe and pair to communicate between threads

    1. Pipe can be open by any threads, because it's only os FDs.
    2. If pipe is open read/write, it's associated by ST, so we MUST free it by the same thread.
    3. If open pipe in one thread, it's ok to free it directly, without close pipe.
    4. If open read in a thread, then open write in another thread, user MUST close it correctly.

commit 40e59ef
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 16:16:23 2021 +0800

    Threads-Hybrid: Enable RTC play API with bugs

commit 9f4fbdb
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 12:39:28 2021 +0800

    Threads-Hybrid: Init ST for each threads

    1. ST is thread-local now.
    2. MUST init st for each threads.
    3. Do it as early as possible.

commit 2fa3aca
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 10:22:13 2021 +0800

    Threads-Hybrid: Extract API server and threads.

commit dd7c7ad
Author: winlin <winlin@vip.126.com>
Date:   Sun Apr 4 20:44:07 2021 +0800

    Threads-Hybrid: Refine conf file

commit 246d7f5
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 11:55:10 2021 +0800

    Threads-Hybrid: Research for extern and __thread

commit 1a0456a
Author: winlin <winlin@vip.126.com>
Date:   Fri Apr 9 22:44:56 2021 +0800

    Threads: TODO: Sort the packets, to avoid NACK

commit b11f958
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 31 18:37:39 2021 +0800

    Threads: Support multiple threads with locks, #2188. 5.0.2

    1. Threads-Log: Use thread to write and reopen logs.
    2. Threads-SRTP: Support decrypt RTP by async SRTP.
    3. Threads-RECV: Support dedicate thread to recv UDP packets.
    4. Threads: Support cpu affinity for threads.
    5. Threads-RECV: Drop received packet if exceed max queue size.
    6. Threads: Use coroutine to consume recv/srtp packets.
    7. Threads: Support Circuit-Breaker to work in storms.
    8. Threads-SRTP: Support async decrypt RTCP
    9. Threads-SEND: Support async send UDP packets
    10. Threads-SRTP: Use async encrypt SRTP packet
    11. Threads-SEND/RECV: Bind handler to listener to support multiple ports.
    12. Threads-RECV: Support tunnel for recv-srtp.
    13. Threads-SEND: Support tunnel for srtp-send.
    14. Threads: Support circuit-breaker dying threshold

commit 8cf7ead
Author: winlin <winlin@vip.126.com>
Date:   Tue Apr 6 20:54:36 2021 +0800

    Threads: Enable threads support for openssl.

commit 28d8f1a
Author: winlin <winlin@vip.126.com>
Date:   Wed Apr 7 10:29:18 2021 +0800

    Threads: Support multiple SRTP/SEND/RECV threads.

    1. Move received and cooked packets queue to thread entry, that is, each thread has its own queue.
    2. In RECV thread, push received packet to queue of source thread, in thread listener.
    3. In SRTP thread, push cooked packet to queue of source thread, in asyn SRTP task.
    4. In hybrid thread, directly and only consume the packets of self thread.
    5. Sync between SRTP task by lock.

commit a505b6b
Author: winlin <winlin@vip.126.com>
Date:   Mon Apr 5 16:17:08 2021 +0800

    Threads: Directly use hybrid thread to consume messages.

commit 14bc7bc
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 30 16:38:30 2021 +0800

    Threads: Keep alive when got RTP plaintext

commit e15737f
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 29 17:18:04 2021 +0800

    Threads: Support circuit-breaker dying threshold

commit d6a92cb
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 30 14:14:26 2021 +0800

    Threads-SEND: Support tunnel for srtp-send.

commit d282ccd
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 29 12:02:41 2021 +0800

    Threads-RECV: Support tunnel for recv-srtp.

commit eb7ce7f
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 22 19:09:04 2021 +0800

    Threads-RECV: Reset the cache buffer when copy

commit 1235b33
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 22 18:45:12 2021 +0800

    Threads-SEND/RECV: Bind handler to listener to support multiple ports.

commit 615da26
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 18:52:11 2021 +0800

    Threads-SRTP: Use async encrypt SRTP packet

    1. Async SRTP support protect RTCP.
    2. Send packet ignore when encrypt size is 0.
    3. Callback to send packet if encrypt done.

commit a26b926
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 18:43:08 2021 +0800

    Threads-SRTP: Use async encrypt SRTP packet

    1. Async SRTP support protect RTP.
    2. Send packet ignore when encrypt size is 0.
    3. Callback to send packet if encrypt done.

commit 56ffc28
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 18:03:40 2021 +0800

    Threads-SEND: Support async send UDP packets

    1. Support async send UDP by SrsAsyncSendManager.
    2. Copy UDP packet by SrsAsyncUdpPacket.
    3. Support SrsUdpMuxSocket raw sendto.
    4. Config the async send by async_send.

commit b0800f5
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 16:16:09 2021 +0800

    Threads-SRTP: Support async decrypt RTCP

    1. SrsAsyncSRTP support unprotect_rtcp packet.
    2. Extract on_rtcp_plaintext from on_rtcp.

commit 949c80e
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 14:58:34 2021 +0800

    Threads-RECV: Change UDP recv max size from 6k to 1500 bytes.

commit 198dca5
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 13:48:36 2021 +0800

    Threads: Merge recv and srtp consume to one timer.

commit 57b771a
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 19 11:30:46 2021 +0800

    Threads: Refine variables and do dispose

    1. Rename packets to srtp or received packets.
    2. Add task dispose API, cleanup in future.
    3. If got packets before init AsyncSRTP, return error.
    4. Never free the SRTPTask, dispose it instead.

commit 9a05d24
Author: winlin <winlin@vip.126.com>
Date:   Thu Mar 18 17:33:51 2021 +0800

    Threads: Support Circuit-Breaker to work in storms.

    1. Config the recv queue, drop packet if exceed.
    2. Config the high and critical threshold and pulse of water level.
    3. If critical water level, disable NACK and TWCC.
    4. If high water level, ignore for NACK insert and send.
    5. Support read the CPU of thread.
    6. Refine SrsPps to support r1s sample.

commit 1c90497
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 17 13:53:55 2021 +0800

    Threads: Use coroutine to consume recv/srtp packets.

commit 957034e
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 17 08:08:17 2021 +0800

    Threads: Use thread-local buffer for log

    1. Call SrsThreadPool::setup() in main(),or  each thread starting.
    2. Initialize the thread-local object in SrsThreadPool::setup().
    3. Change shared log buffer to thread-local.

commit 185359f
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 20:29:35 2021 +0800

    Threads-RECV: Show the dropped packets pps.

commit 6fca411
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 20:26:41 2021 +0800

    Threads-RECV: Drop received packet if exceed max queue size.

    1. Print the number of recv/srtp queue packets.
    2. Drop packet if exceed max recv queue size.

commit 9e554ca
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 19:26:44 2021 +0800

    Threads: Support cpu affinity for threads.

    1. Config cpu_affinity in threads.
    2. Default to not set the cpu affinity.
    3. Support set by cpu range 0-63.

commit fa4c9f9
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 11:42:32 2021 +0800

    Threads: Set the threads name display in top.

commit 1c6e941
Author: winlin <winlin@vip.126.com>
Date:   Tue Mar 16 08:30:21 2021 +0800

    Threads-RECV: Refine the stat for SNMP UDP recv/error

    1. Remove the delta of _srs_snmp_udp_stat.
    2. Use _srs_pps_rloss for receive loss rate.
    3. Use _srs_pps_sloss for send loss rate.

commit 85ea39d
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 15 22:51:58 2021 +0800

    Threads-RECV: Support dedicate thread to recv UDP packets.

    1. Use SrsUdpMuxSocket::raw_recvfrom to read, without ST.
    2. Start a UDP recv thread, to recv packets.
    3. Consume UDP packets in RTC server timer.

commit e3686a4
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 15 21:15:28 2021 +0800

    Threads: Fix bug for SRTP and Log thread nanosleep.

commit 88165ba
Author: winlin <winlin@vip.126.com>
Date:   Mon Mar 15 20:36:20 2021 +0800

    Threads-SRTP: Support decrypt RTP by async SRTP.

    1. Create dedicate thread for async srtp.
    2. SrsSecurityTransport use async srtp if config on.
    3. Cook SRTP packets in async srtp.
    4. Consume cooked SRTP packets in RTC server timer.

commit f068540
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 21:39:36 2021 +0800

    Threads-SRTP: Config and add files for the async-srtp

    1. If configed the async srtp, use a new object.
    2. Allow sync and async srtp, by config.

commit 2367483
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 22:11:02 2021 +0800

    Threads-Log: Refine stat for sync wait, in log thread.

commit 6a1d6a0
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 18:43:54 2021 +0800

    Threads-Log: Remove dual queue for sys logs.

    1. It exists delay for multiple threads.
    2. There is overlay for cache of coroutine queues.
    3. Risk when other threads write logs.

commit 6ddbc5f
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 18:30:23 2021 +0800

    Threads-Log: Refine comments for global variable.

    1. Dual queue for async logs, exists risk.
    2. Flush the logs is too slow, because it depends on logs and interval.

commit 6718c38
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 10:42:09 2021 +0800

    Threads-Log: Refine dual queue for log thread.

    1. App/User controls the interval to flush coroutine-queue.
    2. Use srs_update_system_time to get time for log.
    3. Stat the thread sync in us, in SrsThreadPool.
    4. Change default interval for thread to 5s.

commit 37aee44
Author: winlin <winlin@vip.126.com>
Date:   Sun Mar 14 08:55:17 2021 +0800

    Threads-Log: Support dual queue cache for async logs.

    1. Create dual queue, the coroutine queue and thread queue.
    2. The coroutine queue cache logs does not require lock.
    3. When need to flush, flush the logs from coroutine-queue to thread-queue.
    4. Finally, flush thread-queue to disk.

commit 4918160
Author: winlin <winlin@vip.126.com>
Date:   Sat Mar 13 07:09:56 2021 +0800

    Threads-Log: Support thread-safe queue SrsThreadQueue.

    1. Wrap std::vector to thread-safe queue.
    2. Keep API compatible with std::vector.
    3. SrsAsyncFileWriter use thread-safe queue instead.

commit 3810f5f
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 18:59:14 2021 +0800

    Threads-Log: Remove utest for reload log configs.

commit bf95fdf
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 18:48:03 2021 +0800

    Threads-Log: Use thread to write and reopen logs.

    1. Remove support for reload log configs.
    2. Add config for thread pool cycle interval.
    3. Add config for log flush interval.
    4. Create a SrsAsyncLogManager to create writers.
    5. Create a SrsAsyncFileWriter to write file async.

commit 0cea382
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 12:02:36 2021 +0800

    Threads-Log: Refine thread lock type to ERRORCHECK.

    1. Set lock attribute type to PTHREAD_MUTEX_ERRORCHECK.
    2. If dead lock, the pthread_mutex_lock return EDEADLK.
    3. We assert fail if lock failed.

commit 668c9c1
Author: winlin <winlin@vip.126.com>
Date:   Fri Mar 12 08:45:10 2021 +0800

    Threads-Log: Run hybrid server in thread.

    1. Create thread when execute by thread pool.
    2. The primordial thread check all threads status.
    3. Have not complete the cleanup and stop.

commit 831b77b
Author: winlin <winlin@vip.126.com>
Date:   Thu Mar 11 18:06:16 2021 +0800

    Threads-Log: Refine API and main workflow.

    1. Use SrsThreadPool to execute the hybrid server.
    2. Right now, directly run in primordial thread, that is no threads.
    3. Define the main APIs of SrsThreadPool.
@winlinvip winlinvip changed the title Support Multiple-CPUs(or Threads) to improve concurrency. Support Multiple-CPUs(or Threads) to improve concurrency. 支持多线程/多进程/多核。 May 10, 2021
@winlinvip winlinvip added this to the SRS 5.0 release milestone May 10, 2021
winlinvip added a commit that referenced this issue May 19, 2021
commit f4872e5
Author: winlin <winlin@vip.126.com>
Date:   Fri Feb 26 09:13:21 2021 +0800

    ST: For #2188: Remove sendmmsg from ST.

commit aaeb891
Author: winlin <winlin@vip.126.com>
Date:   Thu Mar 11 08:09:54 2021 +0800

    ST: Refine utest script.

commit d1ac9da
Author: winlin <winlin@vip.126.com>
Date:   Wed Mar 3 11:02:25 2021 +0800

    ST: Support fast utest and coverage

commit 8400115
Author: winlin <winlin@vip.126.com>
Date:   Fri Feb 26 07:02:19 2021 +0800

    ST: Always use unserialized accept for linux or darwin

commit c3686f2
Author: winlin <winlin@vip.126.com>
Date:   Fri Feb 26 06:54:05 2021 +0800

    ST: Refine ARFLAGS by disable the verbose log

commit aaa5c4f
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 08:58:46 2021 +0800

    ST: Stack always grows from top to down.

commit dddd466
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 08:51:31 2021 +0800

    ST: Ignore process fork, for single process only

commit 7906cb5
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 08:50:59 2021 +0800

    ST: Fix build warnings

commit d94921b
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 07:27:45 2021 +0800

    ST: Remove select and poll support, only epoll and kqueue

commit 76d2025
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 07:10:47 2021 +0800

    ST: Remove multiple OS support, except Linux and Darwin.

commit 13c4ba3
Author: winlin <winlin@vip.126.com>
Date:   Thu Feb 25 06:59:35 2021 +0800

    ST: Remove __ia64__ CPU support

commit 46c06e4
Author: winlin <winlin@vip.126.com>
Date:   Wed Feb 24 11:37:27 2021 +0800

    ST: Remove unused files for ST
@ChenMoGe2
Copy link

ChenMoGe2 commented Dec 23, 2021

So, does it support multi-threading for WebRTC now? After starting multiple processes and adding port reuse, only one core is used on a multi-core machine.

@winlinvip
Copy link
Member Author

winlinvip commented Feb 19, 2022

I've been working with Node.js for almost a year now, and I found that it is very similar to SRS's coroutine + multithreading, and I can basically see the future of SRS's multithreading.

The simplicity is quite good, which is what we want. We can't refer to Go for multithreading because it is true multithreading, while Node.js's multithreading is actually single-threaded for each thread, without locks and such. Go actually has locks.

Multithreading without thread synchronization is more suitable for maintenance.

I still insist on business separation for multithreading, with the stream still in one thread. This doesn't solve performance issues, but it does solve some CPU-intensive and freezing issues, such as:

  1. Audio transcoding: For example, AAC to Opus, and vice versa.
  2. DNS resolution: Currently implemented using system functions, multithreading can avoid blocking (of course, using UDP to implement it yourself is also a solution, but it's more complicated).
  3. Writing logs: Writing to disk, usually not a problem, but who knows? Experts say that if you can rely on the disk, everything should be on the tree.
  4. HLS, DASH, and DVR: Writing a lot of data to disk, some friends have tested that it has an impact when there are more than 10 channels, because the disk is really unreliable.

After solving these problems, the stability will be improved, and sometimes it is definitely affected by these factors.

I don't want to do multithreading for streams, because from the perspective of ease of use, for example, the API needs to double the maintenance cost for clustering, requires scheduling logic (no matter how simple), increases the steps for troubleshooting, and cannot simply evaluate the load. All these factors will greatly reduce the maintenance level of the entire project.

The only advantage of multithreading for streams is to increase multi-core capabilities, which can be achieved through cascading (which will be supported in the future) and business scheduling. If you think that running one process on one machine is too wasteful, you can use multiple ports or implement it with Pods. In any case, if you have already reached the point of focusing on high performance, it must be a large business volume of tens of thousands or even hundreds of thousands. If such a large business volume does not have research and development capabilities, it will either be a pitfall or a death trap.

@winlinvip winlinvip changed the title Support Multiple-CPUs(or Threads) to improve concurrency. 支持多线程/多进程/多核。 Support Multiple-CPUs(or Threads) to improve concurrency. Jul 18, 2023
@ossrs ossrs locked and limited conversation to collaborators Jul 18, 2023
@winlinvip winlinvip converted this issue into discussion #3665 Jul 18, 2023
@winlinvip winlinvip added EnglishNative This issue is conveyed exclusively in English. TransByAI Translated by AI/GPT. and removed EnglishNative This issue is conveyed exclusively in English. labels Jul 27, 2023

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Labels
Discussion Discussion or questions. Enhancement Improvement or enhancement. TransByAI Translated by AI/GPT.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants